How do Community Edition users find out about mboxadmin

[aniewels]

Would you be so kind? :)

[cjwilber]

Would you be so kind? :)

OK. Here it is. I actually had a few problems with the perl script so I have the following:
a bash script in cron.daily called spamlearn.cron containing

Code: Select all

#!/bin/sh
for user in `/opt/scalix/bin/omshowu -m all -i`
do
date
echo "Running spamlearn for $user"
/usr/local/bin/spamlearner.pl localhost $user Spam INBOX
done 2>>/var/log/spamlearn.log 1>>/var/log/spamlearn.log


This calls the file /usr/local/bin/spamlearner.pl, which contains:

Code: Select all

#!/usr/bin/perl
use strict;
use warnings;
use Mail::IMAPClient;

my $usage =
"ARGS must be :
\targv1 : imap host
\targv2 : imap user
\targv3 : spam mailbox on imap server
\targv4 : ham mailbox on imap server\n";

die($usage) if(@ARGV != 4);
my ($host,$user,$spam,$ham)=@ARGV;

my $imap  = new Mail::IMAPClient( 'Server' => $host , 'User' => 'mboxadmin:adminuser:' . $user , 'Password' => 'password') or die "Unable to login to IMAP $@";

foreach my $folder ($imap->folders) {
        $imap->select($folder) or next;
        if ($folder eq $spam) {
                #For spam fetch all messages because delete them each day
                print "Processing $folder folder\n";
                my @list = $imap->messages or next;
                print "Found " . @list . " messages in the $folder folder\n";
                foreach my $mess (@list){
                                open (MBOX_SPAM, "|spamassassin -d | sa-learn --spam") or die "Can't open pipe: $!";
                                my @output = $imap->fetch(($mess,'RFC822')) or die "Unable to fetch $@";
                                print MBOX_SPAM "$output[1]" if(defined($output[1]));
                                close (MBOX_SPAM);
                }
                ### Remove seen spam messages, because we don't need them anymore
                my $nrDeleted = $imap->delete_message( scalar($imap->seen) ) or warn "Could not delete_message: $@\n";
                print "$nrDeleted messages deleted\n";

                ### Ok, the messages are deleted, but in fact they aren't (welcome to IMAP ;-))
                ### So, we should expunge the folder to actually delete the messages
                $imap->expunge($folder) or die "Could not expunge: $@\n";
        }
        elsif ($folder eq $ham) {
                print "Processing $folder folder\n";
                #This process will affect the status of recent and unseen flags, so take copy before and restore afterwards
                print "Checking for Unseen and Recent messages:\n";
                my @recent = $imap->recent or warn "No recent msgs: $@\n";
                my @unseen = $imap->unseen or warn "No unseen msgs: $@\n";
                print "There are " . @recent . " recent messages, and " . @unseen . " unseen messages in the $folder folder\n";
                print "The status of these messages will be retained.\n";
                #For ham we only fetch a day's worth of messages otherwise we would be continually re-learning same messages
                my $yesterday = time()-86400;
                my @list = $imap->since($yesterday) or warn "search: No emails found since yesterday\n";
                if ($@) {
                        warn "Error in search: $@\n";
                }

                print "Found " . @list . " messages in $folder folder\n";
                foreach my $mess (@list){
                                open (MBOX_HAM,"|spamassassin -d | sa-learn --ham") or die "Can't open pipe: $!";
                                my @output = $imap->fetch(($mess,'RFC822')) or die "Unable to fetch $@";
                                print MBOX_HAM "$output[1]" if(defined($output[1]));
                                close (MBOX_HAM);
                }
                #Restore the Unseen and Recent flags after first checking whether there were any in that state
                if (@unseen > 0 ) {
                        print "Restoring the Unseen status for " . @unseen . " messages in the $folder folder\n";
                        $imap->unset_flag("Seen",@unseen) or warn "Could not reset flag for Unseen messages: $@\n";
                }
                if (@recent > 0 ) {
                        print "Restoring the Recent status for " . @recent . " messages in the $folder folder\n";
                        $imap->set_flag("Recent",@recent) or warn "Could not set flag for Recent messages: $@\n";
                }
        }
}
$imap->disconnect() or die "Unable to disconnect\n";
#Run a final run with --sync option
system("sa-learn --sync") == 0 or die "Could not sync bayes database: $?\n";
print "Spamassassin learning of $ham and $spam folders for user $user finished\n";
print "\n";

replace 'password' and 'adminuser' as apprropriate
Hope this helps a few people.
Thanks to others on the list who I heavily plagiarised to create this script.

[jch]

Sorry about the partially documented -i option. It's fixed for the next major release now, it might make its way into a future patch.

If you use IPC::Open2 instead of a socket then you can use authnc to avoid having passwords anywhere although it can be a pain to make sure you've got a valid kerberos ticket present. Everyone should configure Kerberos though: it's very cool and we'll be relying on Kerberos for secure inter-server communications in the future.

jch

[jch]

Oops. Didn't see the perl script -- replacing the socket with an IPC::Open2 is going to be rather tricky.

Personally, I wouldn't bother setting the \Recent flag: its semantics make it pretty useless for almost all clients. It also won't stay set: those same semantics mean that the \Recent lag is cleared when you disconnect. You can also avoid messing with the unseen flag if you use "RFC822.PEEK" instead of "RFC822".

jch

[mephisto]

OK, here's yet another implementation of this, now streamlined using jch's suggestions. I also chose to include the admin password into the perl script. Make sure it has the permissions 0700 to be on the safe side. The script also waits one week until it deletes spam messages giving your colleagues ample time to sort out false positives.

/usr/local/sbin/spamlearner.pl

Code: Select all

#!/usr/bin/perl
use strict;
use warnings;
use Mail::IMAPClient;

my $usage =
"ARGS must be :
\targv1 : imap user (password will be prompted)\n";

die($usage) if(@ARGV != 1);
my ($user) = @ARGV;

my $host = "localhost";
my $adminuser = "sxadmin";
my $password = 'password';
my $spam = "Junk-E-Mail";
my $ham = "INBOX";

my $imap  = new Mail::IMAPClient( 'Server' => $host , 'User' => "mboxadmin:$adminuser:$user" , 'Password' => $password  ) or die "Unable to connect to imap server\n";

print "Scanning user $user\n";

foreach my $folder ($imap->folders) {
        $imap->select($folder) or next;
        if ($folder eq $spam) {
                #For spam fetch all messages older than one week because we want to delete them each day
                print "Processing $folder folder\n";
                my $lastweek = time()-604800;
                my @all = $imap->messages or next;
                $imap->unset_flag("Seen",@all) or warn "Could not reset all messages to unseen: $@\n";
                my @list = $imap->before($lastweek) or next;
                print "Found " . @list . " messages in the $folder folder older than 1 week\n";
                foreach my $mess (@list){
                                open (MBOX_SPAM, "|spamassassin -d | sa-learn --spam -u $user ") or die "Can't open pipe: $!";
                                my @output = $imap->fetch(($mess,'RFC822')) or die "Unable to fetch $@";
                                print MBOX_SPAM "$output[1]" if(defined($output[1]));
                                close (MBOX_SPAM);
                }
                ### Remove seen (=checked) spam messages, because we don't need them anymore
                my $nrDeleted = $imap->delete_message( scalar($imap->seen) ) or warn "Could not delete_message: $@\n";
                print "$nrDeleted messages deleted\n";

                ### Ok, the messages are deleted, but in fact they aren't (welcome to IMAP ;-))
                ### So, we should expunge the folder to actually delete the messages
                $imap->expunge($folder) or die "Could not expunge: $@\n";
        }
        elsif ($folder eq $ham) {
                print "Processing $folder folder\n";
                #For ham we only fetch a day's worth of messages otherwise we would be continually re-learning same messages
                my $yesterday = time()-86400;
                my @list = $imap->since($yesterday) or warn "search: No emails found since yesterday\n";
                if ($@) {
                        warn "Error in search: $@\n";
                }

                print "Found " . @list . " messages in $folder folder\n";
                foreach my $mess (@list){
                                open (MBOX_HAM,"|spamassassin -d | sa-learn --ham -u $user ") or die "Can't open pipe: $!";
                                my @output = $imap->fetch(($mess,'RFC822.PEEK')) or die "Unable to fetch $@";
                                print MBOX_HAM "$output[1]" if(defined($output[1]));
                                close (MBOX_HAM);
                }
        }
}
$imap->disconnect() or die "Unable to disconnect\n";

print "Spamassassin learning of imap folders $ham and $spam finished\n";


This runs once a day and only checks the mailboxes of regular users - leaving out ressources and the admin user.

/etc/cron.daily/spamlearner.sh

Code: Select all

#!/bin/sh
scalixusers=`/opt/scalix/bin/omsearch -e ' ! RESOURCE-TYPE=1' -s -m UL-AUTHID|awk -F "=" '{ print $2 }'|grep -v admin`;
for i in $scalixusers ; do
        /usr/local/sbin/spamlearner.pl $i >> /var/log/spamlearner.log 2>> /var/log/spamlearner.log
        sa-learn --sync -u $i 2> /dev/null
done

[tmoz]

hi cjwilber,
i was interested in you magic script to learn for spamassassin. maybe you could send me an email? => thomas.mozelt@vmg.at
that would be great! thanks in advance.
thomas

[leigh]

I presume that an mboxadmin user can't look at another mboxadmin user's mail box?
I've tried this and it seems that if I, as an mboxadmin, attempt to login using "mboxadmin:me:anotheruser" "mypassword", it works OK so long as anotheruser is not also an mboxadmin. If they are, it fails.

[jch]

That's right. It was an interesting bug that arose from the fact that plain authentication and SASL authentication are somewhat intereaved. That mboxadmin stuff is a bit mesy because of the two different mechanisms.

If you can use a SASL mechanism like GSSAPI or DIGEST-MD5 instead of a plain login then you won't trip over the bug. I've just tried authnc with DIGEST-MD5 with both styles of mboxadmin login that I mentioned earlier and it works on server version 10.0.0.175.

jch

[leigh]

Is there any way of tying authnc in with perl's MAIL::IMAPClient? I'm trying something similar to the above post, fetching junk mail and feeding spam and ham into spamassasssin.
I'm using MAIL::IMAPClient to do all the IMAPing, and would rather not trip over incorrectly set mboxadmins.

[cartel]

Does this work for accounts managed from Active Directory? I tried using this method from SWA and Outlook to access another users mailbox from a user granted mboxadmin, however i couldnt login using the login form:

mboxadmin:superman@ENTERPRISE.COM:minion@ENTERPRISE.COM

I also granted mboxadmin to a premium user I created directly in SAC and tried to login to both SWA and Outlook with the following

mboxadmin:admin:minion@ENTERPRISE.COM

which unfortunately failed. I understand I wouldnt have been able to then configure delegation anyway.

For the record with Exchange one can configure Mailbox Rights (to allow login from outlook, and from there one can set permission and delegates) from the ADUC, so I think this would be a useful feature to be able to set login delegates from SAC. You dont need to teach SAC to grok subfolders in this way. Also the "Open these additional mailboxes" feature from Outlook would be a nice to have in SWA.

[jch]

There's nothing special about an AD managed Scalix account that the mboxadmin code even knows about, so, yes, mboxadmin should work for those accounts. Try it with authnc first before trying it with SWA and Outlook. I mentioned, originally, turning audit on and that's useful for checking stuff out.

Outlook/MAPI (as opposed to Outlook/IMAP) needs a different approach that I've not described and I can never remember It's to do with the difficulty of setting up the profile.

The Exchange "mailbox rights" is global delegate access, as least in Scalix server terms and it has been mentioned as a nice-to-have any number of times by some people, but it's never got further than that.

Outlook's "open these additional mailboxes" is handled differently in SWA and IMAP. Other people's mailboxes to which you have been granted access appear underneath "Other People" (or something like that) in both. In SWA there's a mechanism for adding these on a menu somewhere, once they've been added permanently they also appear in IMAP clients. (It's actually done by an IMAP extension and I still can't think why I didn't do it by the IMAP CREATE request).

jch

[TCWardrobe]

I know it may not be encouraged to send issue reports to an old thread but as I want to reach the folks who participated here... so here we are

I use a slightly altered script from this thread to feed spam and ham to spamassassin. Spam feeding seems fine but ham not. This is what I get:

Code: Select all

No recent msgs:
search: No emails found since yesterday
Processing Spam folder
Processing Ham folder
Checking for Unseen and Recent messages:
There are 0 recent messages, and 38 unseen messages in the Ham folder
The status of these messages will be retained.
Found 0 messages in Ham folder
Restoring the Unseen status for 38 messages in the Ham folder
bayes: synced databases from journal in 0 seconds: 192 unique entries (1374 total entries)
Spamassassin learning of Ham and Spam folders for user mschmitt finished


There are 38 mails in the Ham folder but it seems they are not fed into SA. What means "recent messages"? I think there may be the problem.
To the script. As said I altered it a bit so it fits my needs. I need to hand-check the spam and ham mails from my users as they do LOTS of errors there (hard to educate users *sigh*) so I hardcoded the values for just one account, mine. I let the users drop not recognized spam and falsely recognized ham in public folders, I check them manually and drop them in personal folders in my mailbox. I did set up a daily cronjob that runs that script. Here it is (password tuned... a bit :

Code: Select all

#!/usr/bin/perl
use strict;
use warnings;
use Mail::IMAPClient;

my ($host,$user,$spam,$ham)=("localhost", "mschmitt", "Spam", "Ham");

my $imap  = new Mail::IMAPClient( 'Server' => $host , 'User' => $user , 'Password' => 'PaSsWoRd') or die "Unable to login to IMAP $@";

foreach my $folder ($imap->folders) {
        $imap->select($folder) or next;
        if ($folder eq $spam) {
                #For spam fetch all messages because delete them each day
                print "Processing $folder folder\n";
                my @list = $imap->messages or next;
                print "Found " . @list . " messages in the $folder folder\n";
                foreach my $mess (@list){
                                open (MBOX_SPAM, "|spamassassin -d | sa-learn --spam") or die "Can't open pipe: $!";
                                my @output = $imap->fetch(($mess,'RFC822')) or die "Unable to fetch $@";
                                print MBOX_SPAM "$output[1]" if(defined($output[1]));
                                close (MBOX_SPAM);
                }
                ### Remove seen spam messages, because we don't need them anymore
                my $nrDeleted = $imap->delete_message( scalar($imap->seen) ) or warn "Could not delete_message: $@\n";
                print "$nrDeleted messages deleted\n";

                ### Ok, the messages are deleted, but in fact they aren't (welcome to IMAP ;-))
                ### So, we should expunge the folder to actually delete the messages
                $imap->expunge($folder) or die "Could not expunge: $@\n";
        }
        elsif ($folder eq $ham) {
                print "Processing $folder folder\n";
                #This process will affect the status of recent and unseen flags, so take copy before and restore afterwards
                print "Checking for Unseen and Recent messages:\n";
                my @recent = $imap->recent or warn "No recent msgs: $@\n";
                my @unseen = $imap->unseen or warn "No unseen msgs: $@\n";
                print "There are " . @recent . " recent messages, and " . @unseen . " unseen messages in the $folder folder\n";
                print "The status of these messages will be retained.\n";
                #For ham we only fetch a day's worth of messages otherwise we would be continually re-learning same messages
                my $yesterday = time()-86400;
                my @list = $imap->since($yesterday) or warn "search: No emails found since yesterday\n";
                if ($@) {
                        warn "Error in search: $@\n";
                }

                print "Found " . @list . " messages in $folder folder\n";
                foreach my $mess (@list){
                                open (MBOX_HAM,"|spamassassin -d | sa-learn --ham") or die "Can't open pipe: $!";
                                my @output = $imap->fetch(($mess,'RFC822')) or die "Unable to fetch $@";
                                print MBOX_HAM "$output[1]" if(defined($output[1]));
                                close (MBOX_HAM);
                }
                #Restore the Unseen and Recent flags after first checking whether there were any in that state
                if (@unseen > 0 ) {
                        print "Restoring the Unseen status for " . @unseen . " messages in the $folder folder\n";
                        $imap->unset_flag("Seen",@unseen) or warn "Could not reset flag for Unseen messages: $@\n";
                }
                if (@recent > 0 ) {
                        print "Restoring the Recent status for " . @recent . " messages in the $folder folder\n";
                        $imap->set_flag("Recent",@recent) or warn "Could not set flag for Recent messages: $@\n";
                }
        }
}
$imap->disconnect() or die "Unable to disconnect\n";
#Run a final run with --sync option
system("sa-learn --sync") == 0 or die "Could not sync bayes database: $?\n";
print "Spamassassin learning of $ham and $spam folders for user $user finished\n";
print "\n";

I am almost sure the problem is that almost always mails in my ham folder are more than one day old but as I am not a perl junkie I am more than uncertain that I can tweak the right parts. As I read the script I see a difference (at least I think so) between ham and spam. For spam it checks ALL mails (and deletes them afterwards), for ham it just checks the "recent" mails (and deletes nothing). I guess "recent" is as in "mails dated after yesterday", so for my environment not very sensible.
As a side note, if spam is detected, SA sends a report to the user and attaches the headers-unchanged-mail. So if it is a false-positive (not spam) I've instructed the users to drag'n'drop those attachments in the public folder for ham. Maybe SA in this constellation has issues with plain mails without SA-headers too?
I hope someone has some advices.

greets
Michael

[jch]

Disclaimer: I haven't read the perl closely, just enough to isolate what I believe is the mistake here.

The IMAP recency concept sucks A message is marked recent if it hasn't been seen by any IMAP client before. If you only have one client that only has one IMAP connection the \Recent flag makes sense: it identifies messages that have appeared since you last connected. If you have a client that makes multiple connections or you have more than one client connecting then \Recent makes rather less sense. The reason you're seeing no recent messages is because the user has been reading their mail (how unreasonable ).

What you actually need to do here is keep track of the highest UID you have ever seen in the ham folder(s). Message UIDs are always increasing so you can guarantee that any message that has a higher UID since you last looked will be new to you -- those are the new HAM messages. This basically is how clients like Thunderbird update their cached copies of messges.

I confess I have very little idea how to modify your perl script to do this.

jch

[TCWardrobe]

Nice to know another thing that sucks with imap thanks for the input! I think I want to do the exact same with the ham folder as I do with spam: fetch all, learn all, delete all. I realised that all mails in my ham folder are just copies anyway... so I cross my fingers and try to dig into the code on my own... let's hope I do nothing wrong

[TCWardrobe]

With kind help from one on IRC (he did not insist on beeing named, so as I think privacy is more important these days, let him remain anonymous ) it looks like this now (more compact, as ham can be deleted as well):

ATTENTION! IT ACTUALLY DELETES E-MAILS! Make sure you work with copies!

Code: Select all

#!/usr/bin/perl
use strict;
use warnings;
use Mail::IMAPClient;

my ( $host, $user, $spam, $ham ) = ( "localhost", "mschmitt", "Spam", "Ham" );

my $imap = new Mail::IMAPClient(
        'Server'   => $host,
        'User'     => $user,
        'Password' => 'PaSsWoRd'
) or die "Unable to login to IMAP $@";

foreach my $folder ( $imap->folders ) {
        # Only the $spam and $ham folder are of interest
        next unless ( $folder eq $spam || $folder eq $ham);
        $imap->select($folder) or next;

        # Set the mode for sa-learn
        # --spam for the $spam folder, --ham for the $ham folder
        my $mode = ( $folder eq $spam ) ? '--spam' : '--ham';

        print "Processing $folder folder\n";
        my @list = $imap->messages or next;
        print "Found " . @list . " messages in the $folder folder\n";

        # Process each message in folder and send it to sa-learn
        foreach my $mess (@list) {
                open( MBOX_SPAM, "|spamassassin -d | sa-learn $mode" )
                  or die "Can't open pipe: $!";
                my @output = $imap->fetch( ( $mess, 'RFC822' ) )
                  or die "Unable to fetch $@";
                print MBOX_SPAM "$output[1]"
                  if ( defined( $output[1] ) );
                close(MBOX_SPAM);
        }

        # As spam folder is no longer needed and the ham folder only contains copies
        # delete every mail processed
        my $nrDeleted = $imap->delete_message( scalar( $imap->seen ) )
          or warn "Could not delete_message: $@\n";
        print "$nrDeleted messages deleted\n";

        ### Ok, the messages are deleted, but in fact they aren't (welcome to IMAP ;-))
        ### So, we should expunge the folder to actually delete the messages
        $imap->expunge($folder) or die "Could not expunge: $@\n";
}
$imap->disconnect() or die "Unable to disconnect\n";

#Run a final run with --sync option
system("sa-learn --sync") == 0 or die "Could not sync bayes database: $?\n";
print
  "Spamassassin learning of $ham and $spam folders for user $user finished\n";
print "\n";

This may just be for my corner case sensible (if you want / need to hand-check ham and spam and work with copies) but seems to work great so far. I hope this will be useful for others.

greets
Michael