The strange case of the repeating domain name.

[hkphooey]

This is an odd one. I installed spamassassin a couple of days ago, and since then I've been seeing some odd error messages in my /var/log/maillog. Here's one example

Code: Select all


Feb  6 17:39:25 mail sendmail[16233]: l169dMdb016233: Milter add: header: X-Spam-Status: No, score=0.7 required=5.0 tests=HTML_MESSAGE,\n\tHTML_TAG_EXIST_TBODY,INFO_TLD,MISSING_MIMEOLE autolearn=no \n\tversion=3.0.6
Feb  6 17:39:25 mail sendmail[16233]: l169dMdb016233: Milter add: header: X-Spam-Checker-Version: SpamAssassin 3.0.6 (2005-12-07) on mail.maildomain.com
Feb  6 17:39:25 mail sendmail[16232]: l169dFim016232: to=<jack@maildomain.com>, delay=00:00:03, xdelay=00:00:03, mailer=relay, pri=40641, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (l169dMdb016233 Message accepted for delivery)
Feb  6 17:39:29 mail sendmail[16245]: l169dMdb016233: SYSERR(root): MX list for maildomain.com.maildomain.com. points back to mail.maildomain.com
Feb  6 17:39:29 mail sendmail[16245]: l169dMdb016233: to=<jack@maildomain.com.maildomain.com>, delay=00:00:05, xdelay=00:00:03, mailer=esmtp, pri=130832, relay=maildomain.com.maildomain.com., dsn=5.3.5, stat=Local configuration error
Feb  6 17:39:30 mail sendmail[16245]: l169dMdb016233: l169dUdb016245: DSN: Local configuration error
Feb  6 17:39:30 mail sendmail[16245]: l169dUdb016245: to=<Jill@maildomain.com.maildomain.com>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=72264, relay=maildomain.com.maildomain.com., dsn=5.3.5, stat=Local configuration error
Feb  6 17:39:30 mail sendmail[16245]: l169dUdb016245: to=root, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=72264, dsn=2.0.0, stat=Sent
Feb  6 17:39:30 mail sendmail[16245]: l169dUdb016245: l169dUdc016245: return to sender: Local configuration error
Feb  6 17:39:30 mail sendmail[16245]: l169dUdc016245: to=root, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=43288, dsn=2.0.0, stat=Sent
Feb  6 17:47:57 mail sendmail[16285]: l169lth3016285: from=<nobody@aol.com>, size=1259, class=0, nrcpts=1, msgid=<bc2.8d70916.32f9a8c2@aol.com>, proto=SMTP, relay=root@localhost
Feb  6 17:47:59 mail sendmail[16286]: l169lv2s016286: from=<nobody@aol.com>, size=1430, class=0, nrcpts=1, msgid=<bc2.8d70916.32f9a8c2@aol.com>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]



It appears something is repeating the domain name so that jack@maildomain.com becomes jack@maildomain.com.maildomain.com. The message is then rejected and sent back to root as Returned mail.

I've checked out a few configuration files but can't really find anything I can explain this behaviour with. I'm assuming its something to do with the interaction of smtpd and sendmail. Are there any sendmail experts out there who have seen this thing before?

Jim

[hkphooey]

I haven't managed to track this one down yet, and now I'm seeing something even stranger -- emails to external domains are now having the local domain address appended to them and consequently can't be delivered. i.e. when someone at the scalix server at mydomain.com sends an email peter@example.com, I'm getting errors like the following in my maillog.

MX list for example.com.mydomain.com. points back to mail.mydomain.com: 1 Time(s)

Obviously this isn't happening all the time, and I can't figure out the circumstances under which it is occuring. However even if it is happening 5% of the time, this is not acceptable.

Anyone have any ideas before I get rid of spamassassin and roll back to the previous configuration?

As I understand it, without spam assassin, scalix uses its own smtpd to deliver mail. Once you install spamassassin, scalix passes mail to sendmail, which passes it to spamass-milter, which passes it to spamassassin before it is delivered. Is this correct?

Maybe if I can remove spam filtering on the outbound traffic, and only let it run on the incoming traffic, that will solve the problem. Clutching at straws here a little ...

[florian]

this seems like a DNS issue. most likely one of your dns-records is not properly "."-terminated in your dns config file, so that the zone name is appended.

the exact way to troubleshoot this would be to check out the dns server config files for mycompany.com and the respective reverse-lookup domain.

cheers,
Florian.

[kanderson]

What entries (if any) are in /etc/mail/local-host-names.

It's possible for sendmail to rewrite the sender headers. Are you aware of this being turned on?

Kev.

[hkphooey]

What entries (if any) are in /etc/mail/local-host-names.

Just mydomain.com
The fqdn of the server is mail.mydomain.com, and this is correct in the /etc/hosts file.

It's possible for sendmail to rewrite the sender headers. Are you aware of this being turned on?.

Not unless the spamassassin installation turned it on. How would I check this? Is it in the sendmail.mc file?

[hkphooey]

this seems like a DNS issue. most likely one of your dns-records is not properly "."-terminated in your dns config file, so that the zone name is appended.

This is something that suddenly started happening about a week ago, and nothing has changed in the DNS record for over 6 months.

In addition its only happening to some emails, not all of them. Most odd.

I've removed spamassassin, and have removed the offending emails from the sendmail queue. Lets see what happens.

[hkphooey]

OK, so I rolled back the configuration a few days ago. Returned sendmail.mc to its original state, and removed the Relay line from smptd.cfg. Restarted sendmail and smtpd.

But its still happening. The domain of the mailserver is being appended to some emails (around 3-5%) and they are not delivered. This happens seemingly at random to both internal and external emails. I can't replicate it at will, yet it happens several times each day. Very frustrating.

There are a few other problems which started happening simultaneously. I didn't mention them earlier as I didn't want to confuse the issue, but I might as well throw them into the fray now. I'm getting the following errors in logwatch. Once again they started happening as soon as I installed spamassassin, and didn't disappear when I rolled back the configuration.

Code: Select all

 Mail Rejected:
     Data format error: 1 Time(s)
 Mail Deferred:
     421 #4.4.5 Too many connections from your host.:
         To: <xxxxxxx@globe.ap.blackberry.net>: 2 Time(s)
         To: <xxxxxx@globe.ap.blackberry.net>,<yyyyy@csl.ap.blackberry.net>: 1 Time(s)
         To: <xxxxxx@csl.ap.blackberry.net>: 13 Time(s)
         To: <xxxxxxx@csl.ap.blackberry.net>,<yyyy@globe.ap.blackberry.net>: 12 Time(s)
     421 Exceeded allowable connection time, disconnecting.:
         To: <xxxxxx@globe.ap.blackberry.net>: 4 Time(s)
         To: <yyyyy@csl.ap.blackberry.net>,<xxxxx@globe.ap.blackberry.net>: 12 Time(s)
         To: <yyyy@csl.ap.blackberry.net>: 8 Time(s)
     451 #4.1.8 Domain of sender address <realemailaddress@mobileemail.vodafone.es> does not resolve:
         To: <yyyyyyy@csl.ap.blackberry.net>: 5 Time(s)
     451 #4.1.8 Domain of sender address <blahblah@mobileemail.vodafone.net> does not resolve:
         To: <yyyyyy@csl.ap.blackberry.net>: 4 Time(s)
         To: <yyyyyy@csl.ap.blackberry.net>,<xxxx@globe.ap.blackberry.net>: 36 Time(s)
     451 #4.1.8 Domain of sender address <realemail@mobilink.blackberry.com> does not resolve:
         To: <yyyyy@csl.ap.blackberry.net>: 22 Time(s)
         To: <yyyy@csl.ap.blackberry.net>,<xxxxx@globe.ap.blackberry.net>: 7 Time(s)
     Bad file descriptor:
         To: <xxxxxx@globe.ap.blackberry.net>: 7 Time(s)
         To: <yyyyyy@csl.ap.blackberry.net>: 5 Time(s)
         To: <yyyyyy@csl.ap.blackberry.net>,<xxxxxx@globe.ap.blackberry.net>: 5 Time(s)


All these errors seem to happen to blackberry related addresses: a couple of the guys have their email forwarding to their blackberry. I'm not sure if its related to the repeating domain problem, but its just as irritating.

[hkphooey]

This is still happening, but now I have some more information.

I sent out a broadcast message to a group. There are 10 people on the distribution list. 7 of them are on the local domain eg joe@mydomain.com, so they all receive email in their Scalix inboxes. Three of the people have their email forwarded to their blackberries. This is set up using the sxaa --redirect --retain command. The email appears in their Scalix mailboxes, and in two of the three cases is delivered externally OK. In one case, however, the relay server address is changed to append the local domain, mydomain.com, and the email is understandably undeliverable. Here's the log entry

Code: Select all

[root@mail log]# grep l2443bU0018323  maillog
Mar  4 12:03:48 mail sendmail[18323]: l2443bU0018323: from=<pluto-mail@mydomain.com>, size=863, class=0, nrcpts=3, msgid=<29694434.691172981012805.JavaMail.root@mail.mydomain.com>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]

Mar  4 12:03:53 mail sendmail[18348]: l2443bU0018323: SYSERR(root): MX list for csl.ap.blackberry.net.mydomain.com. points back to mail.mydomain.com

Mar  4 12:03:53 mail sendmail[18348]: l2443bU0018323: to=<12348888@csl.ap.blackberry.net>, delay=00:00:14, xdelay=00:00:03, mailer=esmtp, pri=180863, relay=csl.ap.blackberry.net.mydomain.com., dsn=5.3.5, stat=Local configuration error

Mar  4 12:03:56 mail sendmail[18348]: l2443bU0018323: to=<tomBogus@cw.blackberry.net>, delay=00:00:17, xdelay=00:00:03, mailer=esmtp, pri=180863, relay=mx02.bis.na.blackberry.com. [216.9.248.33], dsn=2.0.0, stat=Sent (ok:  Message 394516068 accepted)

Mar  4 12:03:58 mail sendmail[18348]: l2443bU0018323: to=<eric0123@globe.ap.blackberry.net>, delay=00:00:19, xdelay=00:00:02, mailer=esmtp, pri=180863, relay=mx04.bis.ap.blackberry.com. [216.9.247.35], dsn=2.0.0, stat=Sent (ok:  Message 8073729 accepted)

Mar  4 12:03:58 mail sendmail[18348]: l2443bU0018323: l2443wU0018348: DSN: Local configuration error


I really can't understand it. In the first case the relay server is altered so that it sends the mail to the wrong place. In the other two cases, everything is fine. I've checked and re-checked the rules in sxaa. The only difference I can see is that the first email address contains numbers, while the other two contain letters or letters and numbers before the @ sign. Could this make a difference?

Any suggestions on where to look would be gratefully accepted. I'm not a sendmail guru and don't really know where these things are hidden.

Once again, this started happening shortly after I tried to install spamassassin. I'm not sure if its related to that. I have now rolled back to the pre-spamassassin configuration and its still happening. Going slowly nuts with this one ...

[florian]

well, the key line seems to be

Code: Select all

Mar  4 12:03:53 mail sendmail[18348]: l2443bU0018323: to=<12348888@csl.ap.blackberry.net>, delay=00:00:14, xdelay=00:00:03, mailer=esmtp, pri=180863, relay=csl.ap.blackberry.net.mydomain.com., dsn=5.3.5, stat=Local configuration error

It selects a "constructed" hostname as the next-hop relay for this address.

can you try "mail 12348888@csl.ap.blackberry.net" from the command line? if it happens for that as well, then we'd know for sure at least that it happens within sendmail.

Florian.

[hkphooey]

Fast reply! ... and far too late on a Sunday night in Europe ...

OK, so yes, sending mail from the command line seems to work. Finds the right relay address. No problems:

Code: Select all

Mar  5 09:36:40 mail sendmail[21493]: l251ac80021493: from=root, size=172, class=0, nrcpts=1, msgid=<200703050136.l251ac80021493@mail.mydomain.com>, relay=root@localhost

Mar  5 09:36:48 mail sendmail[21494]: l251aef0021494: from=<root@mail.mydomain.com>, size=459, class=0, nrcpts=1, msgid=<200703050136.l251ac80021493@mail.mydomain.com>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]

Mar  5 09:36:48 mail sendmail[21493]: l251ac80021493: to=12348888@csl.ap.blackberry.net, ctladdr=root (0/0), delay=00:00:10, xdelay=00:00:08, mailer=relay, pri=30172, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (l251aef0021494 Message accepted for delivery)

Mar  5 09:36:55 mail sendmail[21500]: l251aef0021494: to=<12348888@csl.ap.blackberry.net>, ctladdr=<root@mail.mydomain.com> (0/0), delay=00:00:07, xdelay=00:00:06, mailer=esmtp, pri=120459, relay=mx04.bis.ap.blackberry.com. [216.9.247.35], dsn=2.0.0, stat=Sent (ok:  Message 8368859 accepted)


If I could find a pattern, then it would be easier to troubleshoot, but it just seems so random.

[florian]

well, i'm in san francisco and working on something else anyway, so it's not THAT bad!

hm. i'd actually like to see the SMTP dialog that happens between Scalix and sendmail.

Are you familiar with tcpdump and wireshark/Ethereal?

You should be capturing the SMTP conversation that goes on when the redirect happens. this will be between unix.out and sendmail on port 25/TCP, i.e. for tcpdump and the raw capture i would be starting something like:

Code: Select all

tcpdump -w smtp.pcap -i lo port 25

(or if running wireshark on linux, do it directly in there)

If not running wireshark directly, i'd then load this file into my windows-wireshark and then look for an SMTP packet, right click on it and follow conversation (or however that option is called). This should provide a cleartext SMTP conversation, bidrectional. I

I'd like to see that.... obviously for a failed one.

Florian.

[hkphooey]

tcpdump, no problem. Thanks for the pointer.

I'll email a copy to your private email address (firstname at scalix dot com, right?) so that I don't have to try to hide all the real email addresses!

[florian]

correct... if we find out what's going on here, I'll stil ask you to post summary or data with private information removed, so that everybody else knows what's going on. too valuable to be hidden.

Florian.

[hkphooey]

OK, that didn't work as expected ...

First of all I sent myself an email to my personal address from SWA. No problems. I received the email and the transaction appeared in the tcpdump file. A corresponding entry was found in /var/log/maillog.

However, when I sent an email from SWA to the group (ten internal users, three of which were additionally forwarded out to blackberries), the mail goes out, and appears in /var/log/maillog, but not in the tcpdump pcap file. Maybe this is because the emails are redirected/forwarded rather than sent direct? My knowledge of scalix/sendmail isn't good enough to answer this question myself.

In addition, the email address that failed this time was not the same as the one in the previous instance. This time it chooses a different blackberry user to reject, still sending two and rejecting one. Total insanity ...

[florian]

OK, getting closer...

what do you see when you

Code: Select all

grep -i smtp /var/opt/scalix/??/webmail/swa.properties

and what do you see for

Code: Select all

lsof -i :24
lsof -i :25
lsof -i :465
lsof -i :587

Cheers,
Florian.