[OM 28879] Bad magic number in an item index record

[PaulHerron]

I'm using Community Edition to support mine and my wife's personal and professional e-mail accounts. We have encountered the following error: [OM 28879] Bad magic number in an item index record

The error followed an unfortunate, unclean shut down of the server (not a file system failure) and has rendered her e-mail box inaccessible via SWA or the Outlook connector. Is this a recoverable error?

I have no problem going to Scalix Support to help with this issue -- in fact, I contacted them already. The response was that I need to have a Support Contract or Support Incident pack before they'll respond. Understandable.

I had planned to upgrade to Small Business - 20 with version 11.4 (in anticipation of native support of OTA ActiveSync support for Windows Mobile devices). This incident may hasten that purchase but, before I plunk down the $$$, can someone advise whether there's a chance my wife's mailstore can be restored?

Thanks!




We are running Scalix 10.0.1.3 on Red Hat Enterprise Linux 4, release 1.

Below is the output of omscan -Aafvx and the omscan logs available through SAC.

OUTPUT OF OMSCAN


[root@apollo ~]# omscan -Aafvx
omscan running on 01.27.08 at 00:55:29.
Host computer : apollo.sagecraftsmen.com.
Fix mode requested.

Last omscan tool run on 01.27.08 at 00:52:46; duration 1 minute(s).
Previous server cycle run on 01.24.08 at 12:11:00; duration 1 minute(s).
Current server cycle not started; service reset or delayed.


Extra check option requested.
Active scan option requested.

Scanning file/dir links .... done.

CAUTION: Scanning of message store has started.
Mounted file/dir links must be maintained during the scan.
VxFS file system must not be reorganized - see omscan(1M).

Checking/Scanning data domain ....
~/data/0000055
~/data/000000c
~/data/000005b
~/data/0000043
~/data/000005r
~/data/000003m
~/data/0000003
~/data/000005c
~/data/000000n
~/data/0000059
~/data/000004i
~/data/0000005
~/data/000004a
~/data/0000054
~/data/000000m
~/data/0000033
~/data/000004h
~/data/000002o
~/data/0000035
~/data/000005a
~/data/0000039
~/data/000001h
~/data/0000016
~/data/0000028
~/data/000004o
~/data/0000044
~/data/0000021
~/data/0000010
~/data/000000e
~/data/000000v
~/data/000001n
~/data/0000045
~/data/000001u
~/data/000000u
~/data/000003r
~/data/000004g
~/data/0000012
~/data/0000031
~/data/000003v
~/data/000005v
~/data/0000036
~/data/0000053
~/data/0000029
~/data/0000056
~/data/000002h
~/data/000002n
~/data/0000018
~/data/000000o
~/data/000002s
~/data/000005m
~/data/000000t
~/data/000003s
~/data/000002r
~/data/000000p
~/data/0000014
~/data/000000l
~/data/000005i
~/data/000004m
~/data/000005f
~/data/0000034
~/data/0000011
~/data/000001k
~/data/000005l
~/data/000001r
~/data/000005q
~/data/0000051
~/data/000004s
~/data/000005u
~/data/0000025
~/data/000003u
~/data/0000050
~/data/0000048
~/data/000003d
~/data/000003k
~/data/000000r
~/data/000002i
~/data/000002u
~/data/000001q
~/data/000003h
~/data/000004n
~/data/000002e
~/data/000004q
~/data/0000038
~/data/0000030
~/data/000000g
~/data/000001j
~/data/000001b
~/data/000003p
~/data/000005e
~/data/0000013
~/data/000000f
~/data/000003i
~/data/000004p
~/data/000001m
~/data/0000057
~/data/000005p
~/data/000004v
~/data/0000060
~/data/000004b
~/data/000000k
~/data/000000b
~/data/000002b
~/data/000003g
~/data/000001a
~/data/0000007
~/data/0000049
~/data/000001e
~/data/000003o
~/data/000003q
~/data/000005g
~/data/000002v
~/data/000001v
~/data/000001c
~/data/000003c
~/data/0000024
~/data/0000027
~/data/000003a
~/data/000002a
~/data/000002c
~/data/000001d
~/data/000003t
~/data/000003b
~/data/000000j
~/data/0000004
~/data/0000026
~/data/000004k
~/data/000000s
~/data/0000001
~/data/000002m
~/data/000004u
~/data/000003e
~/data/000004j
~/data/0000047
~/data/000003n
~/data/000002t
~/data/000005n
~/data/000001l
~/data/000004f
~/data/0000002
~/data/000002p
~/data/0000009
~/data/000004e
~/data/000004t
~/data/0000042
~/data/000005k
~/data/000005o
~/data/000004c
~/data/0000015
~/data/000002g
~/data/0000041
~/data/000005j
~/data/000002d
~/data/0000017
~/data/000003l
~/data/000002k
~/data/000005t
~/data/0000019
~/data/000002j
~/data/000002l
~/data/0000046
~/data/000001t
~/data/0000058
~/data/000004l
~/data/000001f
~/data/0000023
~/data/000005d
~/data/000000d
~/data/0000037
~/data/000001i
~/data/000005s
~/data/000000h
~/data/000000q
~/data/000001o
~/data/000001p
~/data/0000052
~/data/000002q
~/data/0000020
~/data/000001s
~/data/000002f
~/data/000003j
~/data/000000a
~/data/000005h
~/data/000003f
~/data/000001g
~/data/0000006
~/data/0000040
~/data/0000022
~/data/000000i
~/data/000004d
~/data/0000032
~/data/000004r
~/data/0000008
done.
Checking/Scanning bulletin board area .... done.
Checking/Scanning user trays ....
postmaster / apollo, sagecraftsmen/CN=Sage Craftsmen Postmaster
sageadmin / apollo, sagecraftsmen/CN=sageadmin
Paul Herron / apollo, sagecraftsmen/CN=Paul Herron
postmaster / apollo, sagecraftsmen/CN=October11th Postmaster
SageFAX / apollo, sagecraftsmen/CN=SageFAX
sxqueryadmin / apollo, sagecraftsmen/CN=sxqueryadmin
Elizabeth Herron / apollo, sagecraftsmen/CN=Elizabeth Keifer Herron


omscan : [OM 4951]
A serious error has occurred. Please see the log files.
Event logged on 01.27.08 at 00:56:12.
Owner/Context Info : ~/user/g00003c
Additional error info:
omscan : [OM 28879] Bad magic number in an item index record.

done.
Checking/Scanning message lists .... done.
Scanning name directories .... done.
Scanning temp domain .... done.
Checking/Scanning message queues .... done.

Missing children ....

Owner Info : sageadmin / apollo, sagecraftsmen/CN=sageadmin
Parent Container : ~/data/000000v/000fcu9:2, RecNum : 3
Missing Child : ~/data/000000v/000fcua:2
Child Type : Container.
Child Creator : ekeiferherron /internet (RFC-822|ekeiferherron@NorwoodSchool)
Child Subject : Activities for Next Week

Owner Info : sageadmin / apollo, sagecraftsmen/CN=sageadmin
Parent Container : ~/data/000000v/000fcu9:2, RecNum : 4
Missing Child : ~/data/000000v/000fcua:4
Child Type : Transaction file.
Child Creator : MAIL-SYSTEM /CN=MAIL-SYSTEM/S-TX=MAIL-SYSTEM/CN-TX=MAIL-SYSTEM
Child Subject : ********

Disk usage ....

USER NAME IN OUT PDG FCAB DLST WB TOTAL (KB)


Bulletin Board area - - - - - - 1

postmaster /apollo,sagecraftsm 1 1 1 12 1 1 17
sageadmin /apollo,sagecraftsme 31 1 1 14 1 31 79
Paul Herron /apollo,sagecrafts 101122 2 6134 13846 1 2096 123201
postmaster /apollo,sagecraftsm 1 1 1 12 1 1 17
SageFAX /apollo,sagecraftsmen/ 505 1 336 15 1 13 871
sxqueryadmin /apollo,sagecraft 1 1 1 1 1 0 5
[root@apollo ~]#




[root@apollo ~]# omscan -Avfx -U "Elizabeth Herron"
omscan running on 01.27.08 at 00:52:46.
Host computer : apollo.sagecraftsmen.com.
Fix mode requested.

Last omscan tool run on 01.27.08 at 00:42:47; duration 0 minute(s).
Previous server cycle run on 01.24.08 at 12:11:00; duration 1 minute(s).
Current server cycle not started; service reset or delayed.


Extra check option requested.
Active scan option requested.

Scanning file/dir links .... done.

CAUTION: Scanning of message store has started.
Mounted file/dir links must be maintained during the scan.
VxFS file system must not be reorganized - see omscan(1M).

Checking/Scanning user trays ....Elizabeth Herron / apollo, sagecraftsmen/CN=Eli zabeth Keifer Herron


omscan : [OM 4951]
A serious error has occurred. Please see the log files.
Event logged on 01.27.08 at 00:52:47.
Owner/Context Info : ~/user/g00003c
Additional error info:
omscan : [OM 28879] Bad magic number in an item index record.

done.



OMSCAN LOGS FROM SAC


Warning. Bad log file record found in file 1 at 0x0032d71a,
skipped 2854 bytes to next record.

ERROR Omscan Server (Omscan Tool ) 01.26.08 22:12:36
[OM 4948] Internal file error. File is ~/user/g00003c/0000001:1.

User Name: Elizabeth Herron / apollo, sagecraftsmen/CN=Elizabeth Keifer Herron


ERROR Omscan Server (Omscan Tool ) 01.26.08 22:12:36
[OM 28879] Bad magic number in an item index record.
-> ct_ReadHeadPart2
<- ct_ReadHeadPart2
-> ct_ExtFileExists
-> ct_OpenExtFile 3002
<- ct_OpenExtFile 3002
-> ct_CloseExtFile
<- ct_CloseExtFile
<- ct_ExtFileExists
-> ct_ReadContentRec 3002 0
<- /build/10.0.1.3/src/lib/ombase/sfl/sfl_io.c:235[100,28879]
<- /build/10.0.1.3/src/lib/ombase/sfl/sfl_Blcked.c:1697[100,28879]
<- /build/10.0.1.3/src/lib/ct/ct_readc.c:273[100,28879]
<- /build/10.0.1.3/src/lib/ct/ct_readc.c:85[100,28879]
<- /build/10.0.1.3/src/bin/scn/scn_ctr.c:1240[100,28879]
<- /build/10.0.1.3/src/bin/scn/scn_ctr.c:1687[100,28879]


ERROR Omscan Server (Omscan Tool ) 01.26.08 22:46:02
[OM 4948] Internal file error. File is ~/user/g00003c/0000001:1.

User Name: Elizabeth Herron / apollo, sagecraftsmen/CN=Elizabeth Keifer Herron
Current errno value: 2


ERROR Omscan Server (Omscan Tool ) 01.26.08 22:46:02
[OM 28879] Bad magic number in an item index record.
Current errno value: 2
-> ct_ReadHeadPart2
<- ct_ReadHeadPart2
-> ct_ExtFileExists
-> ct_OpenExtFile 3002
<- ct_OpenExtFile 3002
-> ct_CloseExtFile
<- ct_CloseExtFile
<- ct_ExtFileExists
-> ct_ReadContentRec 3002 0
<- /build/10.0.1.3/src/lib/ombase/sfl/sfl_io.c:235[100,28879]
<- /build/10.0.1.3/src/lib/ombase/sfl/sfl_Blcked.c:1697[100,28879]
<- /build/10.0.1.3/src/lib/ct/ct_readc.c:273[100,28879]
<- /build/10.0.1.3/src/lib/ct/ct_readc.c:85[100,28879]
<- /build/10.0.1.3/src/bin/scn/scn_ctr.c:1240[100,28879]
<- /build/10.0.1.3/src/bin/scn/scn_ctr.c:1687[100,28879]


ERROR Omscan Server (Omscan Tool ) 01.27.08 00:42:47
[OM 4948] Internal file error. File is ~/user/g00003c/0000001:1.

User Name: Elizabeth Herron / apollo, sagecraftsmen/CN=Elizabeth Keifer Herron
Current errno value: 2


ERROR Omscan Server (Omscan Tool ) 01.27.08 00:42:47
[OM 28879] Bad magic number in an item index record.
Current errno value: 2
-> ct_ReadHeadPart2
<- ct_ReadHeadPart2
-> ct_ExtFileExists
-> ct_OpenExtFile 3002
<- ct_OpenExtFile 3002
-> ct_CloseExtFile
<- ct_CloseExtFile
<- ct_ExtFileExists
-> ct_ReadContentRec 3002 0
<- /build/10.0.1.3/src/lib/ombase/sfl/sfl_io.c:235[100,28879]
<- /build/10.0.1.3/src/lib/ombase/sfl/sfl_Blcked.c:1697[100,28879]
<- /build/10.0.1.3/src/lib/ct/ct_readc.c:273[100,28879]
<- /build/10.0.1.3/src/lib/ct/ct_readc.c:85[100,28879]
<- /build/10.0.1.3/src/bin/scn/scn_ctr.c:1240[100,28879]
<- /build/10.0.1.3/src/bin/scn/scn_ctr.c:1687[100,28879]

[yeak]

I have the same situation like you. The mailbox of some users are corrupted due to power trip, unclean shutdown. We cannot simply delete the mailbox and start over because the user has all their mails since 5 years ago. We just recently migrated from MDaemon to Scalix.

I am not sure if deleting the mailbox and start over will fix it.

Do update your progress, whether you have obtained support from Scalix and fixed, or found other ways.

Thanks.

[PaulHerron]

Yeak,

I did obtain help from Scalix for this problem. I gave them remote access to the server and they were at work in no time. They were able to restore access to much of the mailbox. Regrettably, however, the Inbox store was not recoverable and all messages stored in the Inbox were lost.

I can't say exactly what they did to restore the mailbox but, they were quick.

A couple of lessons from this incident (besides running regular back-ups of the server):

1. If your users are going to store messages for long periods of time, recommend that they file those messages in folders that are not children of Inbox. This helps "spread the risk" so to say so that a corrupt Inbox doesn't result in a total loss.

2. If you are using the Outlook connector ((and presuming you're users don't have sensitive or classified data in their mailstores), use synchronization on the clients to create replicas of the user's mailstore on their computer. You can restore the users mailstore on the server from this replica, if necessary.

3. Make sure your users shut down their computers each evening.

4. Run OMSCAN periodically to keep your mailstore healthy.

[mikevl]

Hi

The problem seems to revolve around you indexs from the logs

first try

sxmkindex -r /mailnode this will sort out indexs for all users on a mailnode.

Try this first

Mike

[joey4712]

We just bought Small Business with 20 users and now have the following problem with one of our users mailboxes:

When running

Code: Select all

omscan -Avfx -U <username>


the output is

Code: Select all

Fix mode requested.

Last omscan tool run on 03.09.08 at 00:07:27; duration 1 minute(s).
Previous server cycle run on 02.09.08 at 21:03:19; duration 125 minute(s).
Current server cycle not started; service reset or delayed.


Extra check option requested.
Active scan option requested.

Scanning file/dir links .... done.

CAUTION: Scanning of message store has started.
         Mounted file/dir links must be maintained during the scan.
         VxFS file system must not be reorganized - see omscan(1M).

Checking/Scanning user trays ....USERNAME / scalix/CN=USERNAME
Check Size : Should be 277034 instead of 277030 for item ~/data/000001l/0000nkd:1.
Check Size : Should be 718274 instead of 718258 for item ~/data/0000006/0000p9q:1.


omscan : [OM 4951]
A serious error has occurred.  Please see the log files.
Event logged on 03.09.08 at 00:21:55.
Owner/Context Info : ~/user/g00003k
Additional error info:
omscan : [OM 28879] Bad magic number in an item index record.

 done.


The log

Code: Select all

omshowlog -p 20


says

Code: Select all

ERROR                          Omscan Server (Omscan Tool   ) 03.09.08 00:21:55
[OM 28879] Bad magic number in an item index record.
Current errno value: 2
        <- ct_ScanExtension
        <- ct_ScanAllExtFields
        -> scn_GetRecItem
        <- scn_GetRecItem
        <- scn_ScanContentRec
        -> ct_ReadNextContentRec
        <- /build/11.4.0/src/lib/ombase/sfl/sfl_io.c:256[100,28879]
        <- /build/11.4.0/src/lib/ombase/sfl/sfl_Blcked.c:1697[100,28879]
        <- /build/11.4.0/src/lib/ct/ct_readc.c:345[100,28879]
        <- /build/11.4.0/src/lib/ct/ct_fndnxt.c:81[100,28879]
        <- /build/11.4.0/src/lib/ct/ct_readc.c:224[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:1304[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:598[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:1304[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:2111[100,28879]


And I also tried to use sxmkindex:

Code: Select all

sxmkindex -r 0 <username>/scalix


And it also says:

Code: Select all

USERNAME /scalix: oops: Bad magic number in an item index record.
Total messages processed: 14562


What can we do? Do we have to request expensive support?


We are currently Using Scalix 11.4.0

[schmoe90]

Use the -S option in omscan to try and fix the size issues, then see what's left of your user's account. It's not looking good though...

[joey4712]

Thank you for your quick response.

I tried running omscan with the -S option and it fixed some errors it didn't fix before. But after running for a while, it aborted with the Message "omscan : [OM 28879] Bad magic number in an item index record.".

I also tried dumping the mailbox with sxmboxexp. But it always aborts with the message "[OM.PM 1004] Internal problem detected, see the log files" when exporting the user's Spam folder.

Is there anything else I can do? Maybe I can delete the Spam folder, which seems to be currupted.

Thanks.

[schmoe90]

I also tried dumping the mailbox with sxmboxexp. But it always aborts with the message "[OM.PM 1004] Internal problem detected, see the log files" when exporting the user's Spam folder.

What do the log files say?

Maybe I can delete the Spam folder, which seems to be currupted.

What makes you think it's corrupted? Sure, delete it, it's just spam anyway, right?

[joey4712]

Thank you for your help and your quick response ...

I also tried dumping the mailbox with sxmboxexp. But it always aborts with the message "[OM.PM 1004] Internal problem detected, see the log files" when exporting the user's Spam folder.

What do the log files say?

Omshowlog says the following:

Code: Select all

ERROR                          Omscan Server (Omscan Tool   ) 04.09.08 16:56:09
[OM 28879] Bad magic number in an item index record.
        <- ct_ScanExtension
        <- ct_ScanAllExtFields
        -> scn_GetRecItem
        <- scn_GetRecItem
        <- scn_ScanContentRec
        -> ct_ReadNextContentRec
        <- /build/11.4.0/src/lib/ombase/sfl/sfl_io.c:256[100,28879]
        <- /build/11.4.0/src/lib/ombase/sfl/sfl_Blcked.c:1697[100,28879]
        <- /build/11.4.0/src/lib/ct/ct_readc.c:345[100,28879]
        <- /build/11.4.0/src/lib/ct/ct_fndnxt.c:81[100,28879]
        <- /build/11.4.0/src/lib/ct/ct_readc.c:224[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:1304[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:598[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:1304[100,28879]
        <- /build/11.4.0/src/bin/scn/scn_ctr.c:2111[100,28879]


Maybe I can delete the Spam folder, which seems to be currupted.

What makes you think it's corrupted? Sure, delete it, it's just spam anyway, right?

When using the tools sxmboxexp or sxmboxlist, both tools abort when they reach the spam folder. It wasn't possible to delete the folder via IMAP or Webmail. I now successfully removed the spam folder by using Outlook with the Scalix Connector. So now let's see what happens when running omscan or sxmboxexp again.

Sxmboxlist runs without errors now. So I hope that deleting the Spam solved the problem ... I will now try to run omscan and sxmboxexp again and hopefully they also complete without errors now.

[joey4712]

Hi again,

Mailbox repaired successfully!

I could get everything working. The following steps were necessary in my case:
- Repair the mailbox with omscan -SAvfx -U USERNAME (so I could open it with Outlook Connector again)
- Delete the Spam folder from Outlook (sxmboxexp and sxmboxlist always aborted when they reached the Spam folder, deleting the folder from IMAP or Webmail was not possible, I had to use Outlook with the Scalix Connector)
- Run another omscan (don't know if this was necessary)
- Export the mailbox with sxmboxexp
- Re-Import the mailbox with sxmboximp

Now the mailbox runs without any errors.


Thank you for your help!