Umlauts missing in certain text-only mails

[mephisto]

Hi,

I'm running a German Outlook 2003 with Scalix 10 and Scalix Connector 10. I migrated my old mails using imapsync. Now some of my old mails lack all high-ascii characters even though they are displayed correctly in SWA. Those are specifically Mails that were sent as text only in thunderbird and then synced to scalix via imapsync. Mails that are displayed correctly are:

- synced HTML-formatted mails
- synced Text-Only Mails sent using other email clients
- Text-Only Mails from any client (incl. Thunderbird) received directly into Scalix

Any hint on how to resolve this?

BTW: if I right click a message that has been synced via IMAP and select "options" the field for the header lines is empty.

Thanks,

Mephisto

[mephisto]

Another umlaut issue: mails sent with charset ISO-8859-15 get their high ascii characters in the subjects replaced with *
Here are the header lines:

Code: Select all

MIME-Version: 1.0
Content-Type: text/plain;

   charset="ISO-8859-15";

   format="flowed"
Content-Transfer-Encoding: 8bit
Content-Disposition: inline

Mails with the charset ISO-8859-1 or UTF-8 are displayed fine.

[mephisto]

Is this being investigated?

[jch]

Well, I can tell you that the non-ASCII characters in an ISO-8859-15 content are being replaced by a * because the support for iso-8859-15 is a little spotty. We know why and its being fixed and with luck it'll all work in the next major release. I don't know when that is though, I'm only engineering and I know nothing :-)

If you can persuade (or hack) imapsync to replace iso-8859-15 with iso-8859-1 you'll be OK so long as you don't rely on the handul of characters that differ between those to character sets (French OE ligatures, the euro symbol and something Finnish I can't immediately spot). If you can leave the subjects as iso-8859-1 and convert the bodyparts to UTF-8 you'll be in even better shape. I know it's not a very good workaround, but I can't think of anything better right now.

jch

[mephisto]

So even if my data store would remain at the current state (loads of ISO-8859-15 mails) then the next release will resolve my umlaut issue? I already synced two mailboxes, and they are currently in production use.

[mephisto]

I wrote this crude script to deal with the messages on the old server right before migration. It's not a fix, but merely a incomplete workaround. It converts the message text of messages with ISO-8859-15 to UTF-8, while the headers are merely "relabled" ISO-8859-1.
There are some caveats however:

- a Euro sign (€) in the subject will be replaced by a *

- filenames of attachments with high ascii characters will look strange. This is because they will remain encoded at ISO-8859-1 while the message is UTF-8. Thunderbird supports this, but Scalix doesn't.

- Cryptographically signed messages will become invalid (thats ok, because the signatures get converted into regular attachments anyway)

Code: Select all

#!/bin/sh

for i in `find . -name "*mail*" -exec grep -li "ISO-8859-15" {} \;`
do
        cp -a "$i" "$i.sedsave"
        sed -e "s/charset=ISO-8859-15/charset=UTF-8/g" \
        -e "s/ISO-8859-15/ISO-8859-1/g" \
        -e "s/iso-8859-15/iso-8859-1/g" \
        "$i.sedsave" > "$i.sedtemp"
        iconv -f ISO-8859-15 -t UTF-8 "$i.sedtemp" > "$i"
        touch -r "$i.sedsave" "$i"
        echo $i
        rm -f "$i.sedsave" "$i.sedtemp"
done

[jch]

I've just been looking into what happens in a little more detail. The problem really is to do with the way that IMAP APPEND handles iso-8859-15 encoded text. What's weird is that if I mail myself a message that's in iso-8859-15 everything is OK and when I go poking around the message store everything is as it should be. It's just that when you APPEND a message with IMAP the body is converted to iso-8859-1 for Outlook clients but it looks OK for IMAP clients (including SWA).

The good news is that you won't have lost anything for IMAP clients, the bad news is that it won't get better for Outlook clients. If the optionn is open to you sync'ing by sending messages and then re-filing will almost certainly be a better solution.

I've raised a bug for this as there's something very odd going on.

jch