TB/TB-2008-03-PXFOLDER
TB -> TB-2008-03-PXFOLDER
Contents
Overview
In today's business environment, working with information in an efficient way has become more important than ever. The Internet with ability to deliver data feeds of various kinds has become a central source of information. As many knowledge workers use their email clients as the primary application on their desktops, it becomes desirable and relevant to be able to integrate such external information into the same environment.
Many of today's email and calendaring systems allow the user to tap into RSS feeds and external calendars following the WebCal standard. The integration point, however, is typically on the client side. With Scalix' Clients-of-Choice paradigm, such a solution favors data redundancy, a hard to manage distributed configuration and a poor user experience.
This is where Scalix Proxy Folders, introduced in Scalix 11.4, comes to help. With this feature, you are able to connect to such external data sources on the server side. A mapping is applied that presents the feed as a standard email or calendar folder, automatically available to all types of supported clients.
In the initial release, Proxy Folders support the aforementioned data types:
Future releases of Scalix will allow other external data to be mapped as well.
Examples for working with Proxy Folders
The following items list typical usage scenarios when working with Proxy Folders. In Scalix 11.4, all administration is done on the command line. Future versions of Scalix will allow end users to setup their own proxy folders from within their Scalix clients.
Adding a RSS Feed for one user
To add an RSS Feed you first need the complete URL to it. For example the BBC London News Feed can be found here:
http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml
To add this feed to user "Feed Demo", with updates performed every 4 hours and located in the users Mailbox under "feeds/london" you would need to run this command:
sxpfadd -n "Feed Demo" -t rss -i 4h -f "feeds/london" \ -u "http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml"
The output from a successful run will look similar to this:
Creating 'feeds/london' From rss feed 'http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml' Proxy Folder created OK: 'feeds/london' (DirectRef: 0001160b79fc8cd8) Populating Proxy Folder from: http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml Proxy Folder Sync start (RSS source) Updating folder 'Feed.Demo@scalix.com/feeds/london' From 'http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml' 0 items currently in folder 0 old item(s) removed 9 new item(s) added Validating updated Proxy Folder contents... Sync completed OK
You should now be able to see this newly created Proxy Folder when running sxpflist
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london
Please see the following SWA screenshot on how the Proxy Folder is displayed in the users mailbox:
Adding a RSS Feed as public folder
To add the freshmeat.net global announcements rss feed located here:
http://feeds.pheedo.com/freshmeatnet_announcements_global
as an public Proxy Folder under feeds/software/freshmeat, updated every 3 hours you would need to run the following command:
$ sxpfadd -p -t rss -i 3h -f "feeds/software/freshmeat" \ -u "http://feeds.pheedo.com/freshmeatnet_announcements_global"
An successful completion of the command will look something like this:
Creating 'Public Folders/feeds/software/freshmeat' From rss feed 'http://feeds.pheedo.com/freshmeatnet_announcements_global' Proxy Folder created OK: 'Public Folders/feeds/software/freshmeat' (DirectRef: 0001000cd78c3e10) Populating Proxy Folder from: http://feeds.pheedo.com/freshmeatnet_announcements_global Proxy Folder Sync start (RSS source) Updating folder 'Public Folders/feeds/software/freshmeat' From 'http://feeds.pheedo.com/freshmeatnet_announcements_global' 0 items currently in folder 0 old item(s) removed 70 new item(s) added Validating updated Proxy Folder contents... Sync completed OK
You should now be able to see the newly created Proxy Folder when running sxpflist:
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london 0001000cd78c3e10 SYNCED Public Folders/feeds/software/freshmeat
Please see the attached screenshot showing a above Proxy Folder in Outlook 2007:
Adding a WebCAL calendar for one user
To subscribe to a German Holiday Calendar located here:
webcal://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics
for user "Feed Demo", updated once a day and displayed under webcal/german in the users mailbox you will need to run this command:
$ sxpfadd -n "Feed Demo" -t webcal -i 1d -f "webcal/german" \ -u "http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics"
Please note that instead of webcal:// you will to use http:// as scheme, otherwise the command will fail.
A successful run will produce output similar to this:
Creating 'webcal/argentina' From webcal feed 'http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics' Proxy Folder created OK: 'webcal/germany' (DirectRef: 00010a045cd0828e) Populating Proxy Folder from: http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics Proxy Folder Sync start (WebCAL source) Updating folder 'Feed.Demo@scalix.com/webcal/argentina' From 'http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics' 0 items currently in folder 0 old item(s) removed 63 new item(s) added Validating updated Proxy Folder contents... Sync completed OK
You can see the newly created folder by running sxpflist.
The following screenshot shows the new calendar in SWA:
Operations
Command Summary
The following new commands are introduced:
| sxpfadd | Add a new Proxy Folder |
| sxpfdel | Delete a Proxy Folder |
| sxpfmod | Modify Proxy Folder details |
| sxpflist | List all Proxy Folders |
| sxpfshow | show Proxy Folder details |
| sxpfsync | refresh Proxy Folder contents |
Detailed information on parameters and usage can be found in the corresponding man pages or by calling the commands with a --help parameter.
Status of Proxy Folder
Proxy Folders can be in one of the following states:
| CREATED | It has been created, but not yet populated with data |
| SYNCING | The folder is in the process of syncing at the moment |
| SYNCED | The sync was completed successfully |
| DELETED | The Proxy Folder has been marked for deletion |
| ERROR | An error occurred |
Modifying a Proxy Folder
Operations on Proxy Folders are carried out using their direct ref to identify them. A direct ref is is an unique ID for every object in the system including Proxy Folders. Using sxpflist you can find out the direct refs of all Proxy Folders:
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london 00010a045cd0828e ERROR Feed Demo/webcal/german 0001000cd78c3e10 SYNCED Public Folders/feeds/software/freshmeat 00010010aba9bfc3 SYNCED Feed Demo/webcal/argentina
The numbers on the left are the direct refs for each folder.
Changing the update interval
To change the update interval for the london feed to every 8 hours you would need to run:
$ sxpfmod -d 0001160b79fc8cd8 -i 8h
Proxy Folder info file: /var/opt/scalix/do/s/proxy/00001dr/info
USER_CN=Feed Demo
AUTH_ID=Feed.Demo@scalix.com
FOLDER=feeds/london
CLASS=IPF.Note
DREF=0001160b79fc8cd8
SOURCE_URL=http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml
SOURCE_TYPE=RSS
SYNC_INTERVAL=8h
Delete a Proxy Folder
To delete a Proxy Folder call sxpfdel with it's direct ref:
$ sxpfdel -d 00010010aba9bfc3 Proxy Folder 'webcal/argentina' deleted
You can now see that the folder has been marked deleted:
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london 00010a045cd0828e SYNCED Feed Demo/webcal/german 0001000cd78c3e10 SYNCED Public Folders/feeds/software/freshmeat 00010010aba9bfc3 DELETED Feed Demo/webcal/argentina
After all user access to this folder has ceased it will be deleted completely.
Additional Properties of Proxy Folders
Administration of Proxy Folders is done via command line tools.
They can be re-exported through standard Scalix means.
In some points Proxy Folders behave differently when compared to normal folders:
| Readonly | Proxy Folders are read only. |
| omscan | Proxy Folders will not be scanned by omscan. |
| Deletion | Contents are not attached to Recovery Folder when deleted. |
| Indexed | Proxy Folders are indexed and by that content is included in search results. |
Limitations
The following limitations currently apply to Proxy Folders as delivered with 11.4:
| Admin only | At this moment only an admin can create, edit or delete Proxy Folders, there is no UI for users. |
| No Auth | Currently Feeds requiring authentication are not supported. |
| Characters | Proxy Folders Names may not contain the the following characters (See Bug 17021, 17118): $ % * / # [ ] |
| Max No. | Only the first 100 entries in a feed are being processed the rest is ignored. |
| Aging | Items which are no longer part of the feed will be deleted from the Proxy Folder (See Bug 16965). |
| URL | Only http:// URLs are supported, webcal:// and javascript redirects are not (See Bug #17064). |
| Username | sxpfadd will fail to create a Proxy Folder when the username contains chinese or japanese characters (See Bug #17220). By using the auth-id instead of the username you can work around this issue. |
| Quota | The content of Proxy Folders is included in a users mailbox size (See Bug #17282). |
| sxmboxexp | Right now sxmboxexp will export Proxy Folders (See Bug #17283). |
Scalix is planning to address these issues in later releases.
Troubleshooting
Accessing log data
Beside the two locations listed below Proxy Folders also write log data in ~/logs/daemon.stderr. This is a known bug and will be addressed in the future (see #17249).
Where to find log files for the background service
The Proxy Folder background service writes log files to
~/proxy/log
This log file can maximally contain 1000 entries and is cycled afterwards.
Accessing log data of a specific proxy folder
To access log data you can use the sxpfshow command:
$ sxpfshow -d 0001160b79fc8cd8 -l
Proxy Folder info file: /var/opt/scalix/do/s/proxy/00001dr/info
USER_CN=Feed Demo
AUTH_ID=Feed.Demo@scalix.com
FOLDER=feeds/london
CLASS=IPF.Note
DREF=0001160b79fc8cd8
SOURCE_URL=http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml
SOURCE_TYPE=RSS
SYNC_INTERVAL=8h
2008-04-17 06:38:14 SYNCED
2008-04-17 06:38:08 Proxy Folder created OK: 'feeds/london' (DirectRef: 0001160b79fc8cd8)
2008-04-17 06:38:08 Populating Proxy Folder from: \
http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml
2008-04-17 06:38:08 Proxy Folder Sync start (RSS source)
2008-04-17 06:38:08 pfRSS -f Feed.Demo@scalix.com/feeds/london \
-u http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml \
-w /var/opt/scalix/do/s/proxy/temp/00001dr -d /var/opt/scalix/do/s/proxy/00001dr
2008-04-17 06:38:08 Updating folder 'Feed.Demo@scalix.com/feeds/london'
2008-04-17 06:38:08 From
'http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml'
2008-04-17 06:38:09 06:38:09 URL:
http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml [6088/6088] \
-> "/var/opt/scalix/do/s/proxy/temp/00001dr/data.raw" [1]
2008-04-17 06:38:09 0 items currently in folder
2008-04-17 06:38:09 Adding item: Labour Hoey would help Tory mayor
2008-04-17 06:38:10 Adding item: Truckers in diesel price protest
2008-04-17 06:38:10 Adding item: UK arrests in Tamil Tigers probe
2008-04-17 06:38:10 Adding item: Are electoral slogans working?
2008-04-17 06:38:10 Adding item: Disabled boy awarded £3m payout
2008-04-17 06:38:11 Adding item: Heathrow rapped by US air chief
2008-04-17 06:38:11 Adding item: WWII planes back RAF hero statue
2008-04-17 06:38:11 Adding item: Paddick urges bus driver action
2008-04-17 06:38:12 Adding item: Service marks church restoration
2008-04-17 06:38:12 0 old item(s) removed
2008-04-17 06:38:14 9 new item(s) added
2008-04-17 06:38:14 Validating updated Proxy Folder contents...
2008-04-17 06:38:14 Deleting work files in /var/opt/scalix/do/s/proxy/temp/00001dr
2008-04-17 06:38:14 Sync completed OK
Alternatively you can also view use sxpfshow to retrieve the location of the folder on disk:
$ sxpfshow -d 0001160b79fc8cd8 Proxy Folder info file: /var/opt/scalix/do/s/proxy/00001dr/info ...
and use a command like less to view the log
$ less /var/opt/scalix/do/s/proxy/00001dr/log
This logfile will contain a maximum of 10 updates and is also cycled afterwards.
Retrieving status or detailed information of a Proxy Folder
To retrieve the status of a Proxy Folder you can either use sxpflist
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london 00010a045cd0828e SYNCED Feed Demo/webcal/german 0001000cd78c3e10 SYNCED Public Folders/feeds/software/freshmeat
or run sxpfshow with the direct ref of the folder in question as parameter:
$ sxpfshow -d 0001160b79fc8cd8
Proxy Folder info file: /var/opt/scalix/do/s/proxy/00001dr/info
USER_CN=Feed Demo
AUTH_ID=Feed.Demo@scalix.com
FOLDER=feeds/london
CLASS=IPF.Note
DREF=0001160b79fc8cd8
SOURCE_URL=http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/england/london/rss.xml
SOURCE_TYPE=RSS
SYNC_INTERVAL=8h
2008-04-17 06:38:14 SYNCED
Folder created but not populated
Please make sure that you did no use any unsupported characters when creating the folder, see Known Limitations in this document for details.
Triggering a sync
When you get an error while populating the folder with the feed data, for example:
$ sxpfadd -n "Feed Demo" -t webcal -i 1d -f "webcal/german" \
-u "http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics"
Creating 'webcal/german'
From webcal feed 'http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics'
Proxy Folder created OK: 'webcal/german' (DirectRef: 00010a045cd0828e)
Populating Proxy Folder from: \
http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics
Proxy Folder Sync start (WebCAL source)
Updating folder 'Feed.Demo@scalix.com/webcal/german'
From 'http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics'
0 items currently in folder
0 old item(s) removed
73 new item(s) added
Validating updated Proxy Folder contents...
Warning: Msg without X-Scalix-Proxy-Id: 00010a41cd549087 Silvester
54d53
< A9CBEA60-EB2E-4F6A-A56D-E9E7EF7FFA75-SEQ-5@www.aslashv.dreamhost.com
73a73
> <No_Proxy_Id_for_00010a41cd549087>
*** Updated contents don't match external data
Or when see you that there is an error by running sxpflist:
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london 00010a045cd0828e ERROR Feed Demo/webcal/german 0001000cd78c3e10 SYNCED Public Folders/feeds/software/freshmeat 00010010aba9bfc3 SYNCED Feed Demo/webcal/argentina
You can trigger a sync of a folder by running the following command (in this case it is the german holiday webcal):
$ sxpfsync -d 00010a045cd0828e -v Proxy Folder Sync start (WebCAL source) pfWebCAL -f Feed.Demo@scalix.com/webcal/german \ -u http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics \ -w /var/opt/scalix/do/s/proxy/temp/00001e7 \ -d /var/opt/scalix/do/s/proxy/00001e7 -v Updating folder 'Feed.Demo@scalix.com/webcal/german' From 'http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics' 08:04:47 URL:http://www.aslashv.dreamhost.com/davpublic/feste%20Feiertage.ics [19342/19342] \ -> "/var/opt/scalix/do/s/proxy/temp/00001e7/data.ics" [1] 73 items currently in folder Msg already exists: 3202CC3C-E114-4D67-8602-68C160D66623-SEQ-4@www.aslashv.dreamhost.com Msg already exists: 5F679E5F-FD56-4E6D-B424-F9A364A44525-SEQ-4@www.aslashv.dreamhost.com ... Msg already exists: 36C34637-AA9C-420B-B62B-471E450FB2E1-SEQ-3@www.aslashv.dreamhost.com 0 old item(s) removed 0 new item(s) added Deleting work files in /var/opt/scalix/do/s/proxy/temp/00001e7 Sync completed OK
If you run sxpflist again you can see that the feed is now in SYNCED state:
$ sxpflist 0001160b79fc8cd8 SYNCED Feed Demo/feeds/london 00010a045cd0828e SYNCED Feed Demo/webcal/german 0001000cd78c3e10 SYNCED Public Folders/feeds/software/freshmeat 00010010aba9bfc3 SYNCED Feed Demo/webcal/argentina
The sxpfsync command also allows you to reinitialize a Proxy Folder (option -r).
Implementation details
The external data is represented as a message within Scalix folders. It is kept in a server side local cache, thus it is effectively a copy of the remote content which is mapped into a users mailbox.
The content of the cache is stored under ~/proxy.
A background service (pf) is responsible for downloading the feeds, using the external command wget, and updating the Proxy Folders. The service can be switched on separately by calling
$ omon pf
and off with
$ omoff pf
After being downloaded the feeds data is converted and inserted into the Proxy Folder via IMAP. For this a special admin user is used called "Proxy-Folder Admin" who will automatically recreated if accidentally deleted.
You can inspect all temporary files created in the update process by running sxpfsync with the retain option (-D):
$ sxpfsync -d 0001000cd78c3e10 -r -D Proxy Folder Sync start (RSS source) Updating folder 'Public Folders/feeds/software/freshmeat' From 'http://feeds.pheedo.com/freshmeatnet_announcements_global' 73 items currently in folder 73 old item(s) removed 73 new item(s) added Validating updated Proxy Folder contents... Work files retained in /var/opt/scalix/do/s/proxy/temp/00001es Sync completed OK
This includes external data, mapped content and IMAP commands and replies.
