Mailbox Corruption?

[kluss0]

I have a user that cannot log into his mail.

In Outlook using Scalix Connector v10.0.0.352 I get the error:
"Unable to open your default mail folders. An Unexpected error has occured. Reasons for the failure are unknown"

In SWA I get "Incorrect username/password" (I know that the username and password are correct).

When I try to do an ontidyu on his mailbox, I get:

[root@postal ~]# omtidyu -B -u "Tushar Mahata " -M
A fatal error has occurred - see the system error log
Content Record has not been upgraded to current container format.
This command is not allowed in this context
This command is not allowed in this context
This command is not allowed in this context
This command is not allowed in this context
This command is not allowed in this context
This command is not allowed in this context

Any ideas??

Thanks,
Kenny

[kluss0]

I forgot to add the output of omscan. It had errors telling me look at the logs. omshowlog shows this:

ERROR Omscan Server (Omscan Tool ) 03.23.06 09:16:20
[OM 4948] Internal file error. File is ~/user/g00004j/0000001:1.

User Name: Tushar Mahata / postal, aylus/CN=Tushar Mahata
Current errno value: 2


ERROR Omscan Server (Omscan Tool ) 03.23.06 09:16:20
[OM 3541] Content Record has not been upgraded to current container format.

Current errno value: 2
-> scn_GetRecItem
<- scn_GetRecItem
<- scn_ScanCtr
-> scn_ScanIndexFile
<- scn_ScanIndexFile
-> scn_GetRecItem
<- scn_GetRecItem
<- scn_ScanContentRec
-> ct_ReadNextContentRec
<- /build/10.0.0.175/src/lib/ct/ct_readc.c:252[3,3541]
<- /build/10.0.0.175/src/lib/ct/ct_readc.c:273[3,3541]
<- /build/10.0.0.175/src/lib/ct/ct_fndnxt.c:81[3,3541]
<- /build/10.0.0.175/src/lib/ct/ct_readc.c:158[3,3541]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:1240[3,3541]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:1687[3,3541]

[kluss0]

Well, this just keeps getting worse.... I tried to fix things by using omscan: omscan -a -f > omscan.log as suggested in some of the documentation. Now, not only do I still have the original user that can't get into their e-mail because of the unconverted container format, but I now have other users that can't get into their mailboxes because certain folders cannot be opened:


[root@postal 1old]# omtidyu -B -u "Abhinav Goyal" -M
Priming mailbox for MIME access for 'Abhinav Goyal /postal,aylus/CN=Abhinav Goyal'
InTray: 444 messages
Folder 'sip_grp': 835 items
OutTray: 0 messages
10.0.0.175
*** Cannot open Folder: 'Search Area'
10.0.0.175
*** Cannot open Folder: 'Calendar'
10.0.0.175
*** Cannot open Folder: 'Contacts'
10.0.0.175
*** Cannot open Folder: 'Drafts'
10.0.0.175
*** Cannot open Folder: 'Notes'
10.0.0.175
*** Cannot open Folder: 'Tasks'
10.0.0.175
*** Cannot open Folder: 'Trash'
Folder 'Sent': 87 items
Folder 'Journal': 0 items
Tracking Area: 5 messages
List Area: 0 messages
10.0.0.175

Can anyone please help me out here???

Thanks,
Kenny

[Valerion]

If I were in your shoes, I'd make a tar archive of /var/opt/scalix so I can roll back if needed.

I would try the following:

1) omcheck -i and omcheck -d to ensure there's no permission issues.

2) what does a

Code: Select all

stat /var/opt/scalix/user/g00004j/0000001

return? Also, try to see if you can find out whose mailbox it belongs to.

Code: Select all

omshowu -n 'User Name' -f

will show you the mailbox directory for a user. You need to check the g00004j part.

3) Take one of the affected users and see what happens if you run Oulook with the /resetfolders switch. That will force it to re-create the default MAPI folders.

4) You can also use omcontain to see the exact state of the mailbox of an affected user. Dangerous tool, but does give back useful info.

5) You may have to re-create the mailboxes for the affected users. omcpoutu/omcpinu may be able to help you with recovery of those mailboxes.

My impression of this is that some mailbox files got corrupted somehow, maybe hardware related.

[kluss0]

omcheck doesn't show any errors.
stat shows:

[root@postal 7old]# stat /var/opt/scalix/user/g00004j/0000001
File: `/var/opt/scalix/user/g00004j/0000001'
Size: 3722265 Blocks: 7280 IO Block: 4096 regular file
Device: 803h/2051d Inode: 89050975 Links: 1
Access: (0660/-rw-rw----) Uid: ( 100/ scalix) Gid: ( 101/ scalix)
Access: 2006-03-24 08:36:41.031857850 -0500
Modify: 2006-03-24 08:36:41.030858015 -0500
Change: 2006-03-24 08:36:41.030858015 -0500

I tried to recreate one of the users, but I can't delete them. I tried in SAC, and by using omdelu. SAC says that the user cannot be removed, and omdelu says that the user does not exist. When I went to re-create it, it says that the authenticationid is already in use... So now I can't go forward, and I can't go back....

I would roll back, but there are currently only 4 users out of 75 that are having a problem. If I use yesterday's backup, then all un-effected users lose mail. I tried /resetfolders, and it didn't do anything. I wish I knew how to use omcontain, but it's password protected... There is also a command called sxconvertu, but it is completely un-documented....

[kluss0]

I tried to recreate one of the users, but I can't delete them. I tried in SAC, and by using omdelu. SAC says that the user cannot be removed, and omdelu says that the user does not exist. When I went to re-create it, it says that the authenticationid is already in use... So now I can't go forward, and I can't go back....

That part is solved. I had to do an omdelent on the authentication ID. All of the other problems still exist, though.

[kluss0]

I have managed to get most of this cleared up by deleteing the users and recreating them, then using omcpinu to copy in their accounts piece by piece from backups. However, I still have one user that just won't work..... He has a bunch of folders that are inaccessable, and I cannot delete them via Outlook, and I can't log into his account via IMAP or SWA. I was able to import the folders from a backup .pst file, but now I have duplicate folders, and Outlook synchronization via Scalix Connect fails every time. Is there any way to manually delete directories from the commandline using something like omlogon?

[ScalixSupport]

Hi Kenny,

Why can't you log on with IMAP or SWA. What is the error?

Are you familiar with issuing imap commands via telnet?

Send me a message to support at scalix dot com to my attention about omcontain.

Regards,
Don

[kluss0]

Don,

When I try to log in, from SWA I get bad username/passwd and from Thunderbird I get READ errors. omshowlog has *MANY* errors to the effect of:

WARNING IMAP Server Da(IMAP Server Pr) 03.24.06 14:36:22
[SYS 2] No such file or directory
File Name: /var/opt/scalix/data/000005s/000ffr3
User Name: Prasad Dorbala / postal, aylus/CN=Prasad Dorbala
Last Folder: Praveen IT
-> im_ItemRef2FName
<- im_ItemRef2FName
-> sfl_OpenSfl
-> im_OpenItem
-> im_ItemRef2FName
<- im_ItemRef2FName
<- /build/10.0.0.175/src/lib/ombase/os/os_open.c:120[1,2]
<- /build/10.0.0.175/src/lib/ombase/sfl/sfl_base.c:369[1,2]
<- /build/10.0.0.175/src/lib/ct/ct_open.c:688[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:2052[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:3687[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:2052[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:3687[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_syncg.c:1140[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_syncg.c:1803[1,2]

WARNING IMAP Server Da(IMAP Server Pr) 03.24.06 14:38:24
[SYS 2] No such file or directory
File Name: /var/opt/scalix/data/000005s/000ffr3
User Name: Prasad Dorbala / postal, aylus/CN=Prasad Dorbala
Last Folder: Praveen IT
-> im_ItemRef2FName
<- im_ItemRef2FName
-> sfl_OpenSfl
-> im_OpenItem
-> im_ItemRef2FName
<- im_ItemRef2FName
<- /build/10.0.0.175/src/lib/ombase/os/os_open.c:120[1,2]
<- /build/10.0.0.175/src/lib/ombase/sfl/sfl_base.c:369[1,2]
<- /build/10.0.0.175/src/lib/ct/ct_open.c:688[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:2052[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:3687[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:2052[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_prpitm.c:3687[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_syncg.c:1140[1,2]
<- /build/10.0.0.175/src/lib/ual/ual_syncg.c:1803[1,2]

When I run omscan I get:

[root@postal ~]# omscan -Axv -U "Prasad Dorbala"
omscan running on 03.24.06 at 15:25:54.
Host computer : postal.
Report mode requested.

Last omscan tool run on 03.24.06 at 13:27:55; duration 1 minute(s).
Previous server cycle run on 03.23.06 at 18:07:20; duration 18 minute(s).
Current server cycle not started; service reset or delayed.


Extra check option requested.
Active scan option requested.

Scanning file/dir links .... done.

CAUTION: Scanning of message store has started.
Mounted file/dir links must be maintained during the scan.
VxFS file system must not be reorganized - see omscan(1M).

Checking/Scanning user trays ....Prasad Dorbala / postal, aylus/CN=Prasad Dorbala


omscan : [OM 4951]
A serious error has occurred. Please see the log files.
Event logged on 03.24.06 at 15:26:09.
Owner/Context Info : ~/user/g000045
Additional error info:
omscan : [OM 28879] Bad magic number in an item index record.

done.
and in omshowlog:
ERROR Omscan Server (Omscan Tool ) 03.24.06 15:26:09
[OM 4948] Internal file error. File is ~/data/000009a/000g46s:1.

User Name: Prasad Dorbala / postal, aylus/CN=Prasad Dorbala
Current errno value: 2


ERROR Omscan Server (Omscan Tool ) 03.24.06 15:26:09
[OM 4948] Internal file error. File is ~/data/0000094/000f93u:1.

User Name: Prasad Dorbala / postal, aylus/CN=Prasad Dorbala
Current errno value: 2


ERROR Omscan Server (Omscan Tool ) 03.24.06 15:26:09
[OM 4948] Internal file error. File is ~/user/g000045/0000004:1.

User Name: Prasad Dorbala / postal, aylus/CN=Prasad Dorbala
Current errno value: 2


ERROR Omscan Server (Omscan Tool ) 03.24.06 15:26:09
[OM 28879] Bad magic number in an item index record.
Current errno value: 2
-> scn_AddRecItem
<- scn_AddRecItem
<- scn_Mark_REC_ITEM_IN_USE
-> scn_ScanIndexFile
-> ct_GetIndexRecordList1
<- /build/10.0.0.175/src/lib/ombase/sfl/sfl_io.c:235[100,28879]
<- /build/10.0.0.175/src/lib/ombase/sfl/sfl_Blcked.c:1697[100,28879]
<- /build/10.0.0.175/src/lib/ct/ct_ind.c:153[100,28879]
<- /build/10.0.0.175/src/lib/ct/ct_gtolst2.c:233[100,28879]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:1240[100,28879]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:589[100,28879]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:1240[100,28879]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:589[100,28879]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:1240[100,28879]
<- /build/10.0.0.175/src/bin/scn/scn_ctr.c:1687[100,28879]


I can honestly say that I have never done IMAP commands via telnet. SMTP yes. IMAP, no... I'll send off that e-mail about omcontain.

Thanks,
Kenny

[ScalixSupport]

Hi Kenny,

From the output you've posted it looks like these files

/var/opt/scalix/data/000005s/000ffr3

/var/opt/scalix/data/000009a/000g46s

/var/opt/scalix/data/0000094/000f93u

/var/opt/scalix/user/g000045/0000004

aren't found or are somehow corrupted.

Are you seeing anything in /var/log/messages that might indicate disk troubles? What OS is this?

If those files do exist I'm wondering if a spurious lock may be on these files. Sometimes the magic of a reboot cures many ills.

Best wishes,
Don

[kluss0]

Don,

Those files do not exist anywhere. Nothing even remotely resembling a disk issue. This all started with one user getting a corrupted mail box which complained about "Content Record has not been upgraded to current container format" (and I have no doubt that it was self-inflicted, but he won't confess to doing anything.)

After that, I did an omscan -a -f , and that is when everything really started going south. Peoples folders disapeared, mailboxes were corrupt, etc. I could fix almost everyone by deleting mailboxes and re-creating them and populating them with backups. However, this one user is different, since there does not seem to be a good backup of his mailbox (and he is a pretty high-strung VP :-)) So, I can't delete his account. I am stuck trying to repair it rather then just fix the problem with a clean slate..... Oh, this is RH ES4. I bet this wouldn't happen on a Debian system :-)

Thanks,
Kenny

[florian]

omscan is actually a good starting point.

However, you didn't mention the use of the "-A" (capital A) option in your omscan command line; if you don't use -A (Active Mode), omscan actually doesn't do anything, but only reports from the last scheduled run.

You will need something like omscan -Aafvx to get anywhere.

Check the man page for details, cheers,
Florian.

[kluss0]

I have actually run omscan with pretty much every possible combination of options. omcontain will probably do what I need, but given that I can make things a lot worse with it, I need to try it out on a different mailbox first. I may end up firing up Outlook in offline mode, deleting the mailbox and re-creating it, then run a synchronize. I think that that might work, and I can probably get the VP's laptop while he is in a meeting..

Thanks,
Kenny

[ScalixSupport]

I WOULD NOT do that. Export the offline store into a PST file and then reimport that into the new profile.

Cheers,

Sascha.

[kluss0]

Thanks for the heads up. I will do as you suggest. I just want to get this off my plate as fast as possible, since I am not getting good backups of his mailstore at the momnet.

Thanks,
Kenny