Proxy Folders and web proxies

[thecowster]

Proxy Folders are great

But our productive Scalix systems are located in a DMZ, where the firewall restricts all outband web access. To get round this, our IT Security department provides us with a web proxy. This works well with wget.

But "sxpfadd" does not appear to support using a proxy for its online resources.

Is there a way around this? We'd really like to use Proxy Folders on our productive Scalix system.

Cheers
Andy

[florian]

actually i think we use wget internally and it is driven by a script, so you may be able to hack it. there is no official support for a web proxy at this time.

--f

[thecowster]

We have now verified that the firewall is not the issue - we think sxpfadd gets beyond this, and fails on the imap connection. If we request a page that does not exist, we get a 404 response - not a "Connection refused"

Perhaps the "server" that sxpfadd is throwing an error about is in fact the mail server itself.

Full error text is:

Code: Select all

[root@groupware1 downloads]# sxpfadd -vvv -n "Joe Bloggs" -f "TestRSS" -t RSS -u "http://localhost"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work9305
Creating 'TestRSS'
From RSS feed 'http://localhost'
sxtcpc: Can't connect to server: Connection refused
*** Problem running IMAP session for: mboxadmin


If we look in the temporary folder "/var/opt/scalix/ip/s/proxy/temp/work9305" we see a number of files:

Code: Select all

-rw-rw-r-- 1 root scalix 136 Jul 31 14:13 imap.cmds
-rw-rw-r-- 1 root scalix   0 Jul 31 14:13 imap.response
-rw-rw-r-- 1 root scalix  43 Jul 31 14:13 ommodu.err
-rw-rw-r-- 1 root scalix   0 Jul 31 14:13 ommodu.list
-rw-rw-r-- 1 root scalix 554 Jul 31 14:13 omshowu.list
-rw-rw-r-- 1 root scalix 196 Jul 31 14:13 wget.stderr


In the imap.cmds file we have:

Code: Select all

. LOGIN "mboxadmin:mboxadmin:APandy" PF_PWD_STR
. X-SCALIX-ID
. X-CREATE-SPECIAL IPF.Note.Proxy "TestRSS"
. LIST "TestRSS" *
. LOGOUT


..which looks OK.

imap.response is empty (0 bytes) and ommodu.err contains success:

Code: Select all

ommodu: The user was modified successfully


wget.stderr contains what you expect for a successful wget operation:

Code: Select all

--14:13:32--  http://localhost/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 284 [text/html]
200 OK


Finally omshowu.list contains:

Code: Select all

AUTHENTICATION_ID=mboxadmin
GLOBAL_UNIQUE_ID=1c520000b5e50b74-301.791.11.712
USER_NAME=Proxy-Folder Admin /CN=Proxy-Folder Admin
MAILNODE=imap
INTERNET_ADDR="Proxy-Folder Admin" <Proxy-Folder.Admin@interactivedata.com>
SYSTEM_LOGON=55029
PASSWORD=set
ADMIN_CAP=TRUE
MBOXADMIN_CAP=TRUE
LANGUAGE=C
ACCOUNT_STATUS=Unlocked
LAST_SIGNON=07.31.08 02:33:42
CURRENTLY_LOGGED_ON=FALSE
RECEIVE_MAIL=ENABLED
SERVICE_LEVEL=0
EXCLUDE_TIDY=FALSE
RECOVERY_FOLDER_VISIBLE=FALSE
USER_CLASS=Full
SIS_URL=sxidx://imap.interactivedata-ms.com/0c520000b5e50b74-301.791.11.712


Any ideas where the problem lies?

Cheers
Andy

[Richard Hall]

Hi Andy,

The problem seems to be with the server name used to make the IMAP connection locally. The Proxy Folder scripts use the 'hostname' command to get the FQDN for the local server. In your environment this server name is producing a 'Connection refused' response for the connection to the IMAP port.

I suggest that you modify the /opt/scalix/bin/pfadd script an change the line:
OMHOSTNAME=`omeval hostname`
to set OMHOSTNAME to a server name that will work in your environment , maybe ("localhost").

When you have a server name that works OK, i.e. the sxpfadd command works OK, the you should make the same OMHOSTNAME change to all the Proxy Folder scripts:

/opt/scalix/bin/pfadd
/opt/scalix/bin/pfdel
/opt/scalix/bin/pflist
/opt/scalix/bin/pfmod
/opt/scalix/bin/pfRSS
/opt/scalix/bin/pfshow
/opt/scalix/bin/pfsync
/opt/scalix/bin/pfWebCAL
/opt/scalix/bin/sxpfoff
/opt/scalix/bin/sxpfon

I guess we need to make hostname an optional parameter to sxpfadd in a future release.

Hopes this helps,
Richard

[thecowster]

Thanks for the tip!

Now I get an imap response (the imap.response file was empty until I changed pfadd), and can see the (empty) folders have been created on the mail server. Below is the response I get in the imap.response file:

Code: Select all

* OK
. OK LOGIN completed, now connected to imap.interactivedata-ms.com
* X-SCALIX-ID ("name" "Scalix IMAP server" "version" "11.4.1.11855" "display-name" "Proxy-Folder Admin" "smtp-address" "Proxy-Folder.Admin@interactivedata.com" "auth-id" "mboxadmin" "global-unique-id" "1c520000b5e50b74-301.791.11.712" "user-class" "Full")
. OK X-SCALIX-ID completed
. OK X-CREATE-SPECIAL completed
* LIST (\X-Scalix-Proxy \X-FolderClass=IPF.Appointment \X-Owner \X-DirectRef=0006b2a7c277cc7e \X-ModDate=20080804075943) "/" "Public Folders/BBC Radio 4 Program"
. OK LIST completed
* BYE IMAP Server logging out
. OK LOGOUT completed


But the created imap folders are empty, and the sxpfadd command is still failing, but with a different error:

Code: Select all

[root@groupware1 ~]# sxpfadd -vvv -p -t "WebCAL" -f "BBC Radio 4 Program" -i 1d -u "http://homepage.mac.com/leuliett/BBC_Radio4.ics"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work24684
Creating 'Public Folders/BBC Radio 4 Program'
From WebCAL feed 'http://homepage.mac.com/leuliett/BBC_Radio4.ics'
  ** Error: Selected item 'F-0-1486572601' not found
sxmboxlist : Item(s) selected are not present in the archive data
*** Proxy Folder container not created: /00000pf


It looks like wget and imap create work fine - but populating the folder is unsuccessful.

On closer inspection the failure occurs (for public folders) in the call to sxmboxlist. This command already uses the correct imap backend (at least, sxmboxlist --user works fine for me).

The full command is:

Code: Select all

sxmboxlist -d 0006b2a7c277cc7e --debug dref --debug filename


The error this command gives when executed from the command line is:

Code: Select all

  ** Error: Selected item 'F-0-1486572601' not found
sxmboxlist : Item(s) selected are not present in the archive data


However, the response is different if I try to add a proxy folder for an individual user. The sxmboxlist command does not throw an error for users, but the sxpflist command still shows no proxy folders:

Code: Select all

[root@groupware1 work24684]# sxpfadd -vvv -n "Andy Pandy" -f "Radio 4"  -t "WebCAL" -i 1d -u "http://homepage.mac.com/leuliett/BBC_Radio4.ics"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work27627
Creating 'Radio 4'
From WebCAL feed 'http://homepage.mac.com/leuliett/BBC_Radio4.ics'
Proxy Folder created OK: 'Radio 4' (DirectRef: 0006c4037781d156)
Populating Proxy folder from: http://homepage.mac.com/leuliett/BBC_Radio4.ics
No Proxy folders present


Using an imap connection I can see the newly created folders - but all are empty.

Code: Select all

[root@groupware1 work24684]# sxpflist -f
No Proxy folders present


Cheers
Andy

[Richard Hall]

Hi Andy,

Sorry to hear that you are still having problems.

It looks like the Proxy Folder is being created OK now, but the recording of the PF information doesn't occur. (Of course using this Radio_4 URL on my system works fine - I get 400 calendar entries added.)

Are you getting any directories with a name of ~scalix/ip/s/proxy/00nnnnn added? If so, what files do they contain?

Also could you try adding a simple RSS feed to Public Folders and a user mailbox, the BBC weather is a nice small feed for testing:
http://feeds.bbc.co.uk/weather/feeds/rs ... d/2071.xml

I expect this will also fail for you but it may provide a bit of additional information to help with solving your problem.

Cheers - Richard

[thecowster]

Richard,

In the folder ~scalix/ip/s/proxy/ there are only the following folders:

Code: Select all

2jm9cq3  2jm9cqf  2jm9cu7  2jm9cuh  2jm9d90  temp


and these two files:

Code: Select all

log  pid


(The log file only seems to currently log that starting and stopping of the proxy folder service.)

Within the (non-temp) folders I see the following files:

Code: Select all

[root@groupware1 proxy]# ll 2jm9cqf
total 24
-rw-rw---- 1 scalix scalix 3712 Aug  4 10:58 00000pf
-rw-rw---- 1 scalix scalix  279 Aug  4 11:06 info
-rw-rw---- 1 scalix scalix  555 Aug  4 11:06 log
-rw-rw---- 1 scalix scalix    5 Aug  4 10:58 next-id
-rw-rw---- 1 scalix scalix   28 Aug  4 11:06 status
-rw-rw---- 1 scalix scalix   11 Aug  4 10:59 sync-utc


00000pf seems to contain binary data, but

info contains:

Code: Select all

# Proxy Folder info file
# ----------------------
USER_CN=Andy Pandy
AUTH_ID=APandy
CLASS=IPF.Appointment
DREF=0006c40bccaafbb3
SOURCE_URL=http://homepage.mac.com/leuliett/BBC_Radio4.ics
SOURCE_TYPE=WebCAL
SYNC_INTERVAL=1d
CREATED_ON=2008-08-04 11:06:41
CREATED_AS=Radio 4.2


log contains all my sxpfadd attempts:

Code: Select all

2008-08-04 10:58:42 Proxy Folder created OK: 'Radio 4.2' (DirectRef: 0006c40bccaafbb3)
2008-08-04 10:58:42 Populating Proxy folder from: http://homepage.mac.com/leuliett/BBC_Radio4.ics
2008-08-04 11:01:32 Proxy Folder created OK: 'Radio 4.2' (DirectRef: 0006c40bccaafbb3)
2008-08-04 11:01:32 Populating Proxy folder from: http://homepage.mac.com/leuliett/BBC_Radio4.ics
2008-08-04 11:06:41 Proxy Folder created OK: 'Radio 4.2' (DirectRef: 0006c40bccaafbb3)
2008-08-04 11:06:41 Populating Proxy folder from: http://homepage.mac.com/leuliett/BBC_Radio4.ics


status contains

Code: Select all

2008-08-04 11:06:41     CREATED

sync-utc contains

Code: Select all

1217840341

If I try adding the BBC weather feed as a public folder I get the following:

Code: Select all

[root@groupware1 groupware-scripts]# sxpfadd -vvv -p -t RSS -f "BBC Weather" -u "http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work17494
Creating 'Public Folders/BBC Weather'
From RSS feed 'http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml'
cmd: sxmboxlist -d 0006c6107596e7d9 --debug dref --debug filename|grep 0006c6107596e7d9 |cut -f 3 | sed "s/00000pf//"
  ** Error: Selected item 'F-0-1486570048' not found
sxmboxlist : Item(s) selected are not present in the archive data
*** Proxy Folder container not created: /00000pf


(I added the output line starting with "cmd:" in order to report the sxmboxlist command before it fails)

There is now a new folder in ~scalix/ip/s/proxy/, called "2jm9fe0". This folder contains two files:

Code: Select all

00000pf next-id

Where next-id contains

Code: Select all

2000

and 00000pf contains binary data.

Adding the BBC weather feed as a user folder gives the following:

Code: Select all

[root@groupware1 2jm9fe0]# sxpfadd -vvv -n "Andy Pandy" -t RSS -f "BBC Weather" -u "http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work20911
Creating 'BBC Weather'
From RSS feed 'http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml'
cmd: sxmboxlist -d 0006c41bc4911ef2 --debug dref --debug filename|grep 0006c41bc4911ef2 |cut -f 3 | sed "s/00000pf//"
Creating pf info file: /var/opt/scalix/ip/s/proxy/2jm9f7n//info
Proxy Folder created OK: 'BBC Weather' (DirectRef: 0006c41bc4911ef2)
Populating Proxy folder from: http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml
No Proxy folders present


There is now a new folder in ~scalix/ip/s/proxy/, called "2jm9f7n". This folder contains the following files:

Code: Select all

[root@groupware1 2jm9f7n]# ll
total 24
-rw-rw---- 1 scalix scalix 3712 Aug  4 12:55 00000pf
-rw-rw---- 1 scalix scalix  282 Aug  4 12:55 info
-rw-rw---- 1 scalix scalix  197 Aug  4 12:55 log
-rw-rw---- 1 scalix scalix    5 Aug  4 12:55 next-id
-rw-rw---- 1 scalix scalix   28 Aug  4 12:55 status
-rw-rw---- 1 scalix scalix   11 Aug  4 12:55 sync-utc


Where

info contains

Code: Select all

# Proxy Folder info file
# ----------------------
USER_CN=Andy Pandy
AUTH_ID=APandy
CLASS=IPF.Note
DREF=0006c41bc4911ef2
SOURCE_URL=http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml
SOURCE_TYPE=RSS
SYNC_INTERVAL=10m
CREATED_ON=2008-08-04 12:55:23
CREATED_AS=BBC Weather


log contains

Code: Select all

2008-08-04 12:55:23 Proxy Folder created OK: 'BBC Weather' (DirectRef: 0006c41bc4911ef2)
2008-08-04 12:55:23 Populating Proxy folder from: http://feeds.bbc.co.uk/weather/feeds/rss/5day/id/2071.xml


next-id contains

Code: Select all

2000


status contains

Code: Select all

2008-08-04 12:55:23     CREATED


and sync-utc contains

Code: Select all

1217847350


Is this helpful? Can I supply you with any further information?

Cheers
Andy

PS - Prior to these tests I migrated all the prior mention scripts alongside pfadd to use the hard-coded scalix server name for OMHOSTNAME.

[Richard Hall]

Hi Andy,

Thanks for the extra info. I can see one problem immediately.

There is some sloppy scripting where the assumption is that the ~/proxy sub-diretories start with '0' which isn't the always the case. You can easily fix this by changing one line in these files:
/opt/scalix/bin/pfdel
/opt/scalix/bin/pflist
/opt/scalix/bin/pfmod
/opt/scalix/bin/pfshow
/opt/scalix/bin/pfsync

Change:

Code: Select all

NUM_PF=`ls $PROXY_DIR/0* 2>/dev/null | wc -l`

to be:

Code: Select all

NUM_PF=`ls $PROXY_DIR/[0-9]* 2>/dev/null | wc -l`

...and try again - hopefully things should work OK now (fingers crossed). I'll raise a bug for this issue.

Cheers - Richard

[thecowster]

Richard

Thanks - this now works for individual users

However, public folders still fail (and this is what we are really after - we want to provide useful additional calendars to all employees as part of our new mail server rollout, specifically, european bank holiday calendars)

The error is still:

Code: Select all

[root@groupware1 ~]# sxpfadd -vvv -p -t "WebCAL" -f "BBC Radio 4 Program - Test2" -i 1d -u "http://homepage.mac.com/leuliett/BBC_Radio4.ics"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work29061
Creating 'Public Folders/BBC Radio 4 Program - Test2'
From WebCAL feed 'http://homepage.mac.com/leuliett/BBC_Radio4.ics'
cmd: sxmboxlist -d 0006c75e2d99710c --debug dref --debug filename|grep 0006c75e2d99710c |cut -f 3 | sed "s/00000pf//"
  ** Error: Selected item 'F-0-1486568848' not found
sxmboxlist : Item(s) selected are not present in the archive data
*** Proxy Folder container not created: /00000pf


Does this mean that the "public" mail folder somehow does not yet exist on our system? If so, how can we create it?

I tried to login to the "mboxadmin" account using mboxadmin:sxadmin:mboxadmin and got a standard list of (empty) folders, so it looks like the mail folder is correct.

Cheers
Andy

[Richard Hall]

Andy,
You could you try adding a Public Folder from the command line:

Code: Select all

omaddbb -s "Test Folder"

and then try the sxpfadd -p again.

Richard

[thecowster]

Seems that there's no problem adding folders to the public mailbox:

Code: Select all

[root@groupware1 ~]# omaddbb -s "Test Folder"
Bulletin Board added
[root@groupware1 ~]# sxmboxlist -d 0006c75e2d99710c
  ** Error: Selected item 'F-0-1486568848' not found
sxmboxlist : Item(s) selected are not present in the archive data


Or, with sxpfadd we get:

Code: Select all

[root@groupware1 ~]# sxpfadd -vvv -p -t "WebCAL" -f "BBC Radio 4 Program - Test2" -i 1d -u "http://homepage.mac.com/leuliett/BBC_Radio4.ics"
Work directory: /var/opt/scalix/ip/s/proxy/temp/work14596
Creating 'Public Folders/BBC Radio 4 Program - Test2'
From WebCAL feed 'http://homepage.mac.com/leuliett/BBC_Radio4.ics'
cmd: sxmboxlist -d 0006c75e2d99710c --debug dref --debug filename|grep 0006c75e2d99710c |cut -f 3 | sed "s/00000pf//"
  ** Error: Selected item 'F-0-1486568848' not found
sxmboxlist : Item(s) selected are not present in the archive data
*** Proxy Folder container not created: /00000pf


It seems that the direct reference returned in the imap.response file is invalid. But I can see the folders created by sxpfadd in the "public" mailbox, over an imap connection.

imap.response contains:

Code: Select all

[root@groupware1 ~]# cat /var/opt/scalix/ip/s/proxy/temp/work14596/imap.response
* OK
. OK LOGIN completed, now connected to imap.interactivedata-ms.com
* X-SCALIX-ID ("name" "Scalix IMAP server" "version" "11.4.1.11855" "display-name" "Proxy-Folder Admin" "smtp-address" "Proxy-Folder.Admin@interactivedata.com" "auth-id" "mboxadmin" "global-unique-id" "1c520000b5e50b74-301.791.11.712" "user-class" "Full")
. OK X-SCALIX-ID completed
. NO cannot create that mailbox
* LIST (\X-Scalix-Proxy \X-FolderClass=IPF.Appointment \X-Owner \X-DirectRef=0006c75e2d99710c \X-ModDate=20080804123849 \X-Total-Msgs=0 \X-Unseen-Msgs=0) "/" "Public Folders/BBC Radio 4 Program - Test2"
. OK LIST completed
* BYE IMAP Server logging out
. OK LOGOUT completed


[Richard Hall]

Andy,
What do the following commands produce:

Code: Select all

omdref 0006c75e2d99710c

and:

Code: Select all

omlistbbs

- thanks

[thecowster]

omdref:

Code: Select all

[root@groupware1 ~]# omdref 0006c75e2d99710c
BULLETIN AREA                                               : RecNo : ItemNo
  BULLETIN BOARD: BBC Radio 4 Program - Test2               :    11 : -1486568848



omlistbbs:

Code: Select all

[root@groupware1 ~]# omlistbbs
  0 BULLETIN BOARD AREA                              (no owner)
  1  Announcements                                   (no owner)
  2  Company Calendar                                (no owner)
  3  Company Contacts                      Bjoern Barstool
  4  Core IT (UK)                          Mark McDonald /imap
  5  Marks calendar                        Sean Cannary /imap/CN
  6  BBC Radio 4 Program                   Proxy-Folder Admin /
  7  Feeds                                 Proxy-Folder Admin /
  8  BBC Radio 4 Program 2                 Proxy-Folder Admin /
  9  BBC Weather                           Proxy-Folder Admin /
 10  BBC Radio 4 Program - Test2           Proxy-Folder Admin /
 11  Test Folder                                     (no owner)


[Richard Hall]

...and one more... what does:

Code: Select all

sxmboxlist -p

produce? Thx

[thecowster]

I cleaned up some of the folders in the public mailbox (as they were empty anyhow), so below are both sxmboxlist and omlistbbs side by side

sxmboxlist -p

Code: Select all

[root@groupware1 ~]# sxmboxlist -p
# Archive contents : Public Folders
# Archive date     : 2008-08-04 16:35:01

F-0000979032    Announcements
F-0004457493    Company Calendar
F-0004751697    Deleted Items
F-0004771905    Marks calendar
F--1486568032   BBC Radio 4 Program - Test2


omlistbbs

Code: Select all

[root@groupware1 ~]# omlistbbs
  0 BULLETIN BOARD AREA                              (no owner)
  1  Announcements                                   (no owner)
  2  Company Calendar                                (no owner)
  3  Marks calendar                        Sean Keeney /imap/CN
  4  BBC Radio 4 Program - Test2           Proxy-Folder Admin /


The direct reference value that seems to correspond to the "Radio 4" entry above is

Code: Select all

[root@groupware1 proxy]# omdref 0006a16a8184b3f7
BULLETIN AREA                                               : RecNo : ItemNo
  BULLETIN BOARD: BBC Radio 4 Program - Test2               :    13 : -1486568032