Users can't list folder contents in SWA, have IMAP issues

[btisdall]

Pardon the cross-post but:

viewtopic.php?t=8545

Also:

When I try to connect to the user's account using Tbird I can't see the folder contents (except for INBOX which I cleared out recently). After a while a large number of imap processes accumulate for the user & the service becomes unavailable for them. These processes need to be manually killed.

Cheers.

[btisdall]

No thoughts on this anyone?

The user is getting rather disgruntled & I'm thinking of desperate measures such an omcpoutu, delete the account an omcpinu. As a second backup, I'm exporting the user's folders as PSTs, but notice that one folder (the one containing the former contents of the inbox) causes Outlook to crash when doing the export - presumably one or more corrupt msgs?

[mikethebike]

Ben,

try copying each folder out in turn (you don't have to bother creating a file "omcpoutu -n username -Si -f /dev/null")
That may give you a better idea of which folders are corrupt.
You may also get a clue to corrupt messages from the fatal log (I think it may record the actual data file that has problems), or maybe the event log (make sure you have admin logging turned up to about level 9 - "omconflvl admin 9", then omshowlog -p 10 -a admin -l 9 > /tmp/filename)
Check the date and time of that file, and do a strings of the file to get some idea of the message date/time, subject, sender etc, then use ommcontain to do in and zap the message.
It may also be worth running an active omscan on the user with the -vfx option, that may also cure the corruption.

Mick

[btisdall]

Thanks Mike,

the user copied out ok, but I think my original diagnosis was wrong as I'm now aware that the IMAP server is behaving very badly across the board. It seems to make a few connections for each user & go no further - even I with my tiny mailbox can't consistently access all my folders & hit 'service unavailable' quickly if using Tbird.

Very odd, the only change I've made to Scalix in ages is introducing pam_ldap a few weeks ago - I regressed the config for ual.remote just for kicks but as expected that didn't help matters.

[btisdall]

This is some output from omshowlog that might be relevant:

Code: Select all

REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:07
[OM 24070] Debug message for Lab use :
imapSatServer:Connected to server
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:11
[OM 23654] The Default Scalix character set (ISO8859_1) was used as the
character set for this process. No explicit setting was found.
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr

[last msg repeated seven times]

REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:12
[OM.DMON 2105] IMAP connection from 192.168.0.3 being handled by process 5099


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:12
[OM 24070] Debug message for Lab use :
imapCheckAndCreateCore: Could not bind to socket /var/opt/scalix/temp/imap-core.127 Error was: Address already in use
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:12
[OM 24070] Debug message for Lab use :
imapSatelliteCommand: Satellite has died, tidying up...
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:12
[OM 24070] Debug message for Lab use :
imapSatServer:Connected to server
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:12
[OM 821] System error 2 on file /var/opt/scalix/sys/unixmap.gw
File Name: /var/opt/scalix/sys/unixmap.gw
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:12
[OM 821] System error 2 on file /var/opt/scalix/sys/unixmap.gw
File Name: /var/opt/scalix/sys/unixmap.gw
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:33
[OM 23654] The Default Scalix character set (ISO8859_1) was used as the
character set for this process. No explicit setting was found.
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:33
[OM 24145] Attempt to convert to or from unknown character set .
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr
  Last Msg Id: 6832554.11454347018
  Last Msg DirectRef: 0002b7af82fc40af


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:33
[OM 3024] Steering information overridden by the following steering line.
      2130 2133 R
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr
  Last Msg Id: 6832554.11454347018
  Last Msg DirectRef: 0002b7af82fc40af


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:33
[OM 24070] Debug message for Lab use :
imapSatelliteCommand: Satellite has died, tidying up...
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr


REPORT                         IMAP Server Da(IMAP Server Pr) 24.08.07 08:35:33
[OM 24070] Debug message for Lab use :
imapCloseSatConn: Forcing close of folder: Calendar
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr

[florian]

I think removing pam_ldap from the stack might be a good test - a faulty PAM module can easily introduce such trouble, provided that they are shared libraries and have code executing in the imap server processes address space.

Have you tried using om_ldap instead? Can you post your ual.remote PAM config file.

Apart from that the only thing I can quickly think of is doing an omscan and also resetting the users imap-cache and OFS logs, to make sure the IMAP cache is completely clean.

Cheers,
Florian.

[btisdall]

Hi F & many thanks for responding.

I regressed ual.remote to the stock config at the start of my troubleshooting but it didn't help

Code: Select all

auth     required om_auth nullok
account  required om_auth
password required om_auth nullok

(the other pam files still include pam_ldap & it certainly works for me ok for both Scalix & other stuff such as su & sudo).

I've run omscan -Aafvx & fsck'd /var/opt/scalix (there was some minor fs damage which fixed up ok - perhaps I should run omscan again after this?).

IMAP cache... this is a 10.0.5 system so not relevant - right?

[florian]

jup, re-running omscan again can't hurt.

true - no IMAP folder cache in 10.0.5 - but actually that should make the overall system much simpler. hmmmp....

Start with the scan, if this reports nothing we'll possibly need to start IMAP tracing and see where the connections are lost.

Florian.

[btisdall]

No improvement after the last omscan I'm afraid Florian.

Ben.

[gren]

Hi,

I noticed a mention of a message with direct reference :
0002b7af82fc40af
in part of the omshowlog output. If this message is still there, then as root, try :
mime.browse 0002b7af82fc40af

Does that give reasonable output and does the program exit with code 0? (echo $? after running the command).
May also be worth trying :
mime.browse -x 0002b7af82fc40af
which re-constructs more of the output from scratch.

Regards,
Gren.

[btisdall]

Hi Gren,

the output is identical in both cases as is the exit status - 0.

Code: Select all

Date: Wed, 19 Apr 2006 09:18:21 +0100
From: "Juergen Schurr" <Juergen@redress.org>
Message-ID: <6832554.1145434701804.JavaMail.root@redress3.local>
Subject: HRW
X-Scalix-Class: IPM.Appointment
X-Scalix-Appt-Date-Start: 2006-04-19T08:30:00Z
X-Scalix-Appt-Date-End: 2006-04-19T09:00:00Z
X-Scalix-Reminder-Set: false
X-Scalix-Reminder-Minutes-Before-Start: 15
X-Scalix-Meeting-Status: 0
X-Scalix-All-Day-Event: false
X-Scalix-Is-Recurring: false
X-Scalix-Busy-Status: 2
X-Scalix-Appt-Label: 0
MIME-Version: 1.0
Content-Type: multipart/alternative;
        boundary="2_0_a7_119543MHTML_=_01"
Content-ID: <0002b7af82fc40af.1@redress3.local>

--2_0_a7_119543MHTML_=_01
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline



--2_0_a7_119543MHTML_=_01
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline


--2_0_a7_119543MHTML_=_01
Content-Type: text/calendar;
        name="meeting.ics";
        method=PUBLISH;
        charset="UTF-8"
Content-Transfer-Encoding: 8bit

BEGIN:VCALENDAR
CALSCALE:GREGORIAN
PRODID:-//Scalix Corporation//Scalix Server 10.0.5.3//EN
VERSION:2.0
METHOD:PUBLISH
BEGIN:VEVENT
UID:6832554.1145434701804.JavaMail.root(a)redress3.local
LAST-MODIFIED:20060419T081821Z
DTSTAMP:20060419T081821Z
DTSTART:20060419T083000Z
DTEND:20060419T090000Z
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
PRIORITY:5
TRANSP:OPAQUE
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
SEQUENCE:0
SUMMARY:HRW
X-SCALIX-LABEL:0
CLASS:PUBLIC
END:VEVENT
END:VCALENDAR

--2_0_a7_119543MHTML_=_01--

Cheers.

[btisdall]

Regarding this error (see above):

Code: Select all

System error 2 on file /var/opt/scalix/sys/unixmap.gw

This file doesn't exist on the system in question - should it, or is the error misleading?

This is version 10.0.5

[btisdall]

Hmmm, a backup of ~scalix from a while back doesn't contain this file so I can only assume the error msg isn't significant (???)

[gren]

Sorry that mime.browse didn't help narrow down the problem
the unixmap.gw file does not exist in a vanilla Scalix installation, so that is probably unrelated to your problem.

When you get a large number of IMAP processes for a single user, could you send a SEGV signal to one or more of the processes and then see if anything useful shows up in omshowlog output. I would expect some sort of stack trace output. with OM 10270 and 10272 errors reported.

For example, if one of the IMAP processes has PID 466, do :

Code: Select all

kill -SIGSEGV 466

Exercise extreme care with the kill command

Regards,
Gren

[btisdall]

Hi Gren & thanks for your help so far. I attempted to list various folders belonging to the user (some did, some did not) & waited until the imap connections maxed out. I then shutdown imap & sent the signal to one of the remaining processes - this is the resulting omshowlog output:

Code: Select all

REPORT                         Administration(omoff         ) 28.08.07 19:34:15
[OM 10522] Subsystem IMAP Server Daemon shutdown by root.


SERIOUS ERROR                  IMAP Server Da(IMAP Server Pr) 28.08.07 19:35:24
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 10526
Procedure trace follows:
  <- ml_CheckPid
  -> ml_CheckPid
  <- ml_CheckPid
  <- sfl_CloseItem
  -> im_ItemRef2FName
  -> FormItemName
  -> im_FormDomainName
  <- im_FormDomainName
  -> os_ul2str
  <- os_ul2str
  <- FormItemName
  <- im_ItemRef2FName
  -> os_GetActualFName
  <- os_GetActualFName
  -> os_GetActualFName
  <- os_GetActualFName
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr
  Last Msg Id: 14820403.1177854351
  Last Msg DirectRef: 00042ecdf3aa8823


SERIOUS ERROR                  IMAP Server Da(IMAP Server Pr) 28.08.07 19:35:24
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xb6)[0xa9ae76]
/opt/scalix/lib/libom_er.so[0xa9b162]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0xa9b31f]
[0x58f420]
/opt/scalix/lib/libom_mdc.so(mdc_wait+0x28)[0x5df3c8]
/opt/scalix/lib/libom_os.so(os_wait+0x1d)[0x61a4dd]
/opt/scalix/lib/libom_cvr.so(cvr_DoBlockedConvert+0x63c)[0xe9bf6c]
/opt/scalix/lib/libom_cvr.so(cvr_VarBlockedConvert+0x241)[0xea2121]
/opt/scalix/lib/libom_cvr.so(cvr_Rtf2Html+0x43c)[0xe9acc4]
/opt/scalix/lib/libom_msl.so[0x1e8519]
/opt/scalix/lib/libom_msl.so(msl_recurseTree_+0x70)[0x1e82c0]
/opt/scalix/lib/libom_msl.so(msl_recurseTree_+0x4a)[0x1e829a]
/opt/scalix/lib/libom_msl.so(msl_recurseTree_+0x4a)[0x1e829a]
/opt/scalix/lib/libom_msl.so(msl_recurseTree_+0x4a)[0x1e829a]
/opt/scalix/lib/libom_msl.so(msl_transform_+0xa8)[0x1ea0e8]
/opt/scalix/lib/libom_msl.so(msl_readContainerOpenedParent+0x3b7)[0x1de3c7]
in.imap41d[0x806246a]
in.imap41d[0x805a6ff]
in.imap41d[0x805fe2c]
in.imap41d[0x8060940]
in.imap41d[0x806121c]
/lib/libc.so.6(__libc_start_main+0xdf)[0x75dd7f]
in.imap41d[0x804da01]
貿à§
User Name: Juergen Schurr / redress3, local/CN=Juergen Schurr
  Last Msg Id: 14820403.1177854351
  Last Msg DirectRef: 00042ecdf3aa8823