Differences

This shows you the differences between two versions of the page.

@@ Line 1: / Line 1: @@
+By: David Lang - davidlang proposed IMAP roadmap (affects non IMAP stuff 2004-05-26 13:22 I'm proposing this as a roadmap becouse I think it is a reasonable path and if we agree on the path I may be able to help implement some of these features
+Among the stated goals for the IMAP interface and popfile are support for multiple IMAP servers support for forking the IMAP processes support for multiple users of popfile
+proposed steps (not nessasarily in order, but I think roughly so)
+  - re-factor the check_for_new messages and reclassiy_on_more (in progress)
+  - restructure the folder lists per my thoughts in IMAP 1.6 (list of known folders with flags rather then seperate lists of watched and output folders)
+  - factor out the move message to new folder to be a seperate routine
+  - factor out the folder selection from the message analysis to the service routine, passing $folder,and $bucket along with $imap
+  - change the initialization, start, stop, and service routines to have a seperate imap connection per folder (change self->imap%%__%% to self->imap%%__%%($folder)
+  - change the server config data to be a table in the database.(login info is part of the server config)
+  - change the folder config/status data to be in the database instead of the config file, refrencing the server config data in each folder entry (includes moving the uid/uidvalidity data out of the config file to the database and updating it more frequently)
+  - change the move_message function to handle multiple servers (detect that the source and destination are not on the same server and download/upload the message instead of doing an IMAP copy). this includes having move_messages maintain it's own set of connections to the servers so that it can send commands without having to worry about what commands are outstanding from the IMAP scanner routines have.
+  - alter the UI to allow multiple servers to be entered in the IMAP config menu and change the folder add option to designate which server the folder is on
+  - at this point it should be simple to allow popfile to folk off the IMAP scanning processes with each one having it's own move_messages routine. (how will database access work at this point??)
+  - seperate the move_messages to it's own process, with the IMAP scanning processes sending it messages to tell it what messages to move where.
+  - change the move_message function to support moving a message from a local file to an IMAP server
+  - change the move_message function to honor the per-bucket modification flags (header alterations)
+  - remove the per-bucket options from the IMAP config menu by adding them to the bucket menu (you would still need to go to the IMAP menu to tell popfile that a particular folder exists, but you would tell popfile that a particular bucket maps to a particular IMAP folder in the bucket menu)
+  - alter the pop3/NNTP proxies to honor the per-bucket IMAP assignment by sending the real message via the move_message function (possibly moved to a seperate module??) and then pass a pre-generated stub message to the client
+  - add additional delivery options to move_message (SMTP, LMTP, NNTP post, local files in various formats, etc)
+  - add additional non-IMAP message source options (POP3 'fetchmail' option as well as the proxy option, SMTP listener, etc) these sources would retreive the messages and then classify them and deliver them via move_message
+thoughts? as long as this is I know I've left something out. (I have remembered it a half a dozen times as I was typeing something else, only to forget it before I finished what I was doing) not to mention things that I have never considered.
+this roadmap ends up morphing popfile into a swis-army-knife classifier
+I start off with tiny concrete steps that are valuble in themselves, but they also build towards the possibility of implementing the later items fairly easily, some of the later options are much more general, and I don't know how nessasary they really will be, some of them are as much for overall consistancy as anythng else (if you say that bucket spam should go to this place, it really should go there, no matter if the source of that spam is via pop, IMAP, or SMTP)
+at the moment popfile is a POP3 proxy that has IMAP bolted onto the side, I haven't seen the info on how the SMTP and NNTP stuff will work so I don't know how well they are really integrated, I think these steps will integrate the IMAP support in nicely and end up opening up a lot of new possibilities
+as the migration towards multi-user support continues limiting the list of allowed folders (and the associated server definitions, which do include login credentials) for a particular user should allow this to co-exist without much conflict between the two projects
+<code>By: David Lang - davidlang
+  RE: proposed IMAP roadmap (affects non IMAP s
+-05-26 13:58
+I remembered one item I had forgotten.
+sometime after item 5, call it 5.5 there are several optimizations that
+can be done to reduce the impact that popfile has on the server.
+today popfile IMAP does a select for each folder on each pass, this
+is not quite as bad as a full login, but it's in the same ballpark (in
+fact the overhead of doing a select on a server is actually significantly
+higher then the act of logging in, until you account for the fact that after
+you login you then have to do a select) after you have a seperate
+connection per folder you can do the select when you login and not need
+to after that.
+along the same lines there are optimizations that could let us avoid the
+need to do a search for each pass, again avoiding a lot of work on the part
+of the server
+  By: Manni Heumann - mannih2001
+  RE: proposed IMAP roadmap (affects non IMAP stuff
+-05-26 14:07
+  That's an amazing roadmap, David.
+  I will have to digest this before I can really comment on it.
+  But here are my first thoughts:
+. I hope that I after the refactoring in (1) and a period of bug-fixing and
+  (yuck) test writing, we will have a version of IMAP that is ready to ship with
+  the next release of POPFile. Once this is out, there will be a lot more bug-
+  fixing, I guess, and I also guess that we will get some feedback that might
+  somehow change the roadmap (I'm not hoping for a changed roadmap, but
+  for the feedback, of course). So I am currently concentrating on (1) and I
+  hope it will be finished some day.
+. Reading your roadmap, one might get the impression that IMAP is the one
+  and only tool for you. I don't know whether I understood the last points on
+  the roadmap correctly, but why would anybody want to have POP3 or NNTP
+  messages delivered to an IMAP server?
+. Any idea about the time scale we are talking about?
+  Manni
+     By: David Lang - davidlang
+     RE: proposed IMAP roadmap (affects non IMAP s
+-05-26 14:32
+     first off I'll answer #3 first, I have no time scale in mind for this,
+     just a list of baby steps that take us towards the goal
+     (I spend so much time at work fighting people who have 'great' ideas
+     that work well in the short term and have to be scrapped in the long term
+     that I am constantly in the mindset of identifying a long term goal and
+     trying to make all changes in the short term support that goal)
+     along the same lines to address #1, getting IMAP to the point where it can
+     ship with the next version is definantly the highest priority goal
+     as for #2, I do use IMAP almost exclusinvly, but I am thinking that there
+     is an oppurtunity for some overall simplification of popfile that will give
+     it additional capabilities if we shuffle the pieces correctly (funnel
+     multiple things through a move_message function so that mixing protocol
+     types is just a matter of figuring out a sane way to configure it)
+     as an example of why you would want to put pop3 messages in an IMAP folder
+     Jim has his main e-mail address with his primary ISP, but he gets a lot of
+     mail (the pesky Lang gene again :-) and can't leave it there. what he
+     currently does is to use popfile to classify the mail, download it to Pegasus,
+     which then sorts the mail based on how popfile has classified it and then
+     uploads the mail to IMAP folders on another server into his 44 different
+     folders (one per bucket)
+     while this process works it is a little more involved the it should need to be.
+     Ideally (and if/when this roadmap is fully implemented) he could tell popfile
+     that it has a data source (watched folder) of type POP3 on server A and popfile
+     would go out, retreive the messages, classify them, and stuff them into the
+     correct IMAP folders.
+     another reason for this sort of thing would be to aggragate multiple e-mail
+     accouunts on different servers into one set of buckets.
+     a reason to go POP3/IMAP source to SMTP destination would be to have popfile
+     have a bucket called 'critical' that you train to recognise critical messages
+     and forward them to your phone/pager. this is far better then giving out your
+     phone/pager e-mail address as popfile will filter the messages so you only get
+     the ones that are really critical
+     think of all the things that people use fetchmail to do, but instead of the
+     complicated fetchmail config language you would have the popfile classification
+     engine to do the decision making for you
+        By: Manni Heumann - mannih2001
+        RE: proposed IMAP roadmap (affects non IMAP s
+-05-26 14:48
+        I have no idea what people are doing with fetchmail, but what you are saying
+        makes a lot of sense. In fact, I would welcome the possibility of merging several
+        of my mail accounts on one good, reliable IMAP server.
+        Manni</code>
+on 6-16 Josh suggested that an additional option for a destination bucket would be to set flags on a message. David Lang agrees that this would be a useful capability

POPFile - Automatic Email Classification

jp:experimentalmodules:imaproadmap

Differences