By: David Lang - davidlang proposed IMAP roadmap (affects non IMAP stuff 2004-05-26 13:22) I'm proposing this as a roadmap because I think it is a reasonable path and if we agree on the path I may be able to help implement some of these features.

Among the stated goals for the IMAP interface and popfile is support for multiple IMAP servers, support for forking the IMAP processes and support for multiple users of popfile.

Proposed steps (not necessarily in order, but I think roughly so):

1. re-factor the check_for_new messages and reclassiy_on_more (in progress)
2. restructure the folder lists per my thoughts in IMAP 1.6 (list of known folders with flags rather then seperate lists of watched and output folders)
3. factor out the move message to new folder to be a seperate routine
4. factor out the folder selection from the message analysis to the service routine, passing $folder,and $bucket along with $imap
5. change the initialization, start, stop, and service routines to have a seperate imap connection per folder (change self→imap__ to self→imap__($folder)
6. change the server config data to be a table in the database.(login info is part of the server config)
7. change the folder config/status data to be in the database instead of the config file, refrencing the server config data in each folder entry (includes moving the uid/uidvalidity data out of the config file to the database and updating it more frequently)
8. change the move_message function to handle multiple servers (detect that the source and destination are not on the same server and download/upload the message instead of doing an IMAP copy). this includes having move_messages maintain it's own set of connections to the servers so that it can send commands without having to worry about what commands are outstanding from the IMAP scanner routines have.
9. alter the UI to allow multiple servers to be entered in the IMAP config menu and change the folder add option to designate which server the folder is on
10. at this point it should be simple to allow popfile to folk off the IMAP scanning processes with each one having it's own move_messages routine. (how will database access work at this point??)
11. seperate the move_messages to it's own process, with the IMAP scanning processes sending it messages to tell it what messages to move where.
12. change the move_message function to support moving a message from a local file to an IMAP server
13. change the move_message function to honor the per-bucket modification flags (header alterations)
14. remove the per-bucket options from the IMAP config menu by adding them to the bucket menu (you would still need to go to the IMAP menu to tell popfile that a particular folder exists, but you would tell popfile that a particular bucket maps to a particular IMAP folder in the bucket menu)
15. alter the pop3/NNTP proxies to honor the per-bucket IMAP assignment by sending the real message via the move_message function (possibly moved to a seperate module??) and then pass a pre-generated stub message to the client
16. add additional delivery options to move_message (SMTP, LMTP, NNTP post, local files in various formats, etc)
17. add additional non-IMAP message source options (POP3 'fetchmail' option as well as the proxy option, SMTP listener, etc) these sources would retrieve the messages and then classify them and deliver them via move_message

Thoughts? As long as this is, I know I've left something out (I have remembered it a half a dozen times as I was typing something else, only to forget it before I finished what I was doing) not to mention things that I have never considered.

This roadmap ends up morphing popfile into a swiss-army-knife classifier.

I start off with tiny concrete steps that are valuble in of themselves, but they also build towards the possibility of implementing the later items fairly easily. Some of the later options are much more general and I don't know how necessary they really will be. Some of them are as much for overall consistancy as anything else (if you say that bucket spam should go to this place, it really should go there, no matter if the source of that spam is via POP3, IMAP, or SMTP).

At the moment, popfile is a POP3 proxy that has IMAP bolted onto the side. I haven't seen the info on how the SMTP and NNTP stuff will work so I don't know how well they are really integrated, but I think these steps will integrate the IMAP support in nicely and end up opening up a lot of new possibilities.

As the migration towards multi-user support continues, limiting the list of allowed folders for a particular user (and the associated server definitions, which do include login credentials) should allow this to co-exist without much conflict between the two projects.

By: David Lang - davidlang
RE: Proposed IMAP Roadmap (affects non IMAP stuff  2004-05-26 13:58)
I remembered one item I had forgotten.

Sometime after item 5, call it 5.5 there are several optimizations that 
can be done to reduce the impact that popfile has on the server.
Today popfile IMAP does a select for each folder on each pass, this 
is not quite as bad as a full login, but it's in the same ballpark (in 
fact, the overhead of doing a select on a server is actually significantly 
higher then the act of logging in, until you account for the fact that after 
you login you then have to do a select). After you have a seperate 
connection per folder, you can do the select when you login and not need
to after that.
Along the same lines, there are optimizations that could let us avoid the 
need to do a search for each pass, again avoiding a lot of work on the part 
of the server.

  By: Manni Heumann - mannih2001
  RE: proposed IMAP roadmap (affects non IMAP stuff  2004-05-26 14:07)
  That's an amazing roadmap, David.

  I will have to digest this before I can really comment on it, 
  but here are my first thoughts:

  1. I hope that after the refactoring in (1) and a period of bug-fixing and 
  (yuck) test writing, we will have a version of IMAP that is ready to ship with 
  the next release of POPFile. Once this is out, there will be a lot more bug-
  fixing, I guess, and I also guess that we will get some feedback that might 
  somehow change the roadmap (I'm not hoping for a changed roadmap, but 
  for the feedback, of course). So I am currently concentrating on (1) and I 
  hope it will be finished some day.

  2. Reading your roadmap, one might get the impression that IMAP is the one 
  and only tool for you. I don't know whether I understood the last points on 
  the roadmap correctly, but why would anybody want to have POP3 or NNTP 
  messages delivered to an IMAP server?

  3. Any idea about the time scale we are talking about?

  Manni

     By: David Lang - davidlang
     RE: proposed IMAP roadmap (affects non IMAP stuff  2004-05-26 14:32)
     
     First off, I'll answer #3 first. I have no time scale in mind for this, 
     just a list of baby steps that take us towards the goal.
     (I spend so much time at work fighting people who have 'great' ideas 
     that work well in the short term and have to be scrapped in the long term 
     that I am constantly in the mindset of identifying a long term goal and 
     trying to make all changes in the short term support that goal.)
     Along the same lines, to address #1, getting IMAP to the point where it can 
     ship with the next version is definitly the highest priority goal.
     As for #2, I do use IMAP almost exclusively, but I am thinking that there 
     is an opportunity for some overall simplification of popfile that will give 
     it additional capabilities if we shuffle the pieces correctly (funnel 
     multiple things through a move_message function so that mixing protocol 
     types is just a matter of figuring out a sane way to configure it).
     As an example of why you would want to put pop3 messages in an IMAP folder,
     Jim has his main e-mail address with his primary ISP, but he gets a lot of 
     mail (the pesky Lang gene again :-) and can't leave it there. What he 
     currently does is to use popfile to classify the mail, download it to Pegasus, 
     which then sorts the mail based on how popfile has classified it and then 
     uploads the mail to IMAP folders on another server into his 44 different 
     folders (one per bucket).
     While this process works, it is a little more involved then it should need to be.
     Ideally (and if/when this roadmap is fully implemented), he could tell popfile 
     that it has a data source (watched folder) of type POP3 on server A and popfile 
     would go out, retreive the messages, classify them and stuff them into the 
     correct IMAP folders.
     Another reason for this sort of thing would be to aggregate multiple e-mail 
     accounts on different servers into one set of buckets.
     A reason to go POP3/IMAP source to SMTP destination would be to have popfile 
     have a bucket called 'critical' that you train to recognize critical messages 
     and forward them to your phone/pager. This is far better then giving out your 
     phone/pager e-mail address as popfile will filter the messages so you only get 
     the ones that are really critical.
     Think of all the things that people use fetchmail to do, but instead of the 
     complicated fetchmail config language you would have the popfile classification 
     engine to do the decision making for you.

        By: Manni Heumann - mannih2001
        RE: proposed IMAP roadmap (affects non IMAP stuff  2004-05-26 14:48)

        I have no idea what people are doing with fetchmail, but what you are saying 
        makes a lot of sense. In fact, I would welcome the possibility of merging several 
        of my e-mail accounts on one good, reliable IMAP server.

        Manni


           On 6-16, Josh suggested that an additional option for a destination bucket 
           would be to set flags on a message. 

               David Lang agrees that this would be a useful capability.