INTRODUCTION It has been close to 2 months since the last POPFile release and boy have I and others been busy. POPFile v0.18.0 is a *major* change from v0.17.9 and you'd do well to read the section entitled ESSENTIAL READING before you install it. This version is backwards compatible with previous versions (even autofixing some errors that the old versions created) but there are some differences. This is also the first release in which two additional developers (additional to me that is) really contributed a lot of code to POPFile. They are Stanley Krute and Sam Schinke. Stan and Sam worked on many of the changes to POPFile (which you can read all about in the CVS logs) including massive changes to the mail parser, whole new skinning system, HTML 4.01 compliance, tons of testing, lots of user interface changes. Guys, *you* made a big difference to POPFile, and I want to say a big "Thank you" in public. This version of POPFile supports unlimited numbers of connections, multiple natural languages, has a much improved UI, has many improvements to help catch spam, has a new Quarantine feature that lets POPFile automatically hide all the mail in a specific bucket (each mail is replaced with a new mail stating that POPFile quarantined a message with an option to see the original), and incorporates features and bug fixes submitted by users. ESSENTIAL READING 1. BACK UP YOUR OLD INSTALLATION: POPFile makes this really easy, just copy the entire POPFile directory somewhere. You can then safely install POPFile v0.18.0 on top of your current installation; I just think a back up is a sensible precaution. 2. YOUR HISTORY WILL CLEAR: I have changed the format of the History files used in this version which means that the old History files will not be read by POPFile and will in fact be deleted to save disk space. If you need to do any reclassifications prior to installing v0.18.0 do them! POPFile does NOT clear buckets, statistics or anything else on install of this version, just the History files. To refresh the History view click the History tab at any time; POPFile will check disk for new messages. 3. IF YOU HAD BROKEN MAGNETS: I have added automatic update of broken magnets. If you had a magnet like [foo] which POPFile mistakenly changed to \[foo\] it should get magically fixed and start working. 4. ACCURACY MIGHT DROP FOR A SHORT WHILE: because of some changes made in the mail parser it is possible that you might see accuracy drop initially and you may find yourself reclassifying a few messages that used to work. This is unfortunate but necessary to make POPFile even more accurate than before and v0.18.0 incorporates changes that make POPFile's classification accuracy better; however old corpuses might need a little retraining. DOWNLOADING You can obtain the latest release of POPFile by visiting http://sourceforge.net/project/showfiles.php?group_id=63137 UPGRADING Just install POPFile on top of the currently installed version. But did you read the ESSENTIAL READING above first? FAQ zonk3r has spent a great deal of time on a POPFile FAQ. Please check it out as it covers many questions that you might have: http://sourceforge.net/docman/display_doc.php?docid=14421&group_id=63137 THE GORY DETAILS 1. Total rearchitecture of the POPFile code into a collection of loadable modules to allow for future expansion of POPFile with different proxy services and different UIs. The major modules are now: - popfile.pl: controlling process that loads the modules that implement POPFile's main functionality; - Proxy::POP3 the original POP3 proxying code from POPFile separated into its own module. In future we should expect to be able to have other Proxy:: modules such as Proxy::IMAP. - UI::HTML contains the current HTML user interface. I hope that we'll add other UI modules such as UI::XMLRPC or UI::SOAP in future. - Classifier::Bayes is the module that does the Naive Bayes classification. (Note this module also includes a method called classify_and_modify that performs the classification and modification of email messages that may be of use to other module writers) POPFile now supports two process types: the top level process round robins between the main modules listed above providing concurrent access to the UI and POP3 proxy. The POP3 proxy itself (see below) forks to provide multiple simultaneous connections. (That last part was originally requested by sschinke) 2. Introduction of a scheme for localization of the user interface using message catalogs stored in the languages/ subfolder. Currently shipping with languages: English, French, German, Danish, Dutch, Brazilian, and (for fun and for testing) H4X0R. Translators can take the English.msg file and translate the strings within to create their own language for the user interface. POPFile detects available languages on start up and they are available from the Configuration page. (Originally requested by pfaelzerchen) 3. Rework of much of the user interface to make pages shorter and clearer. This is mostly limited to look and feel and no items have changed tab in the UI. 4. Total rewrite of the history caching code for clarity with a change from multiple arrays to a single hash containing all the cached information. 5. Automatic update checking now defaults to off and there's a large explanation of what it does in the user interface to allay the fears of many people concerning the possibility that programs like POPFile are spying on them. (Requested by quite a few people) 6. When creating the user interface password the password is now hidden when typing it. (Requested by kraelen, pfaelzerchen) 7. The Proxy::POP3 now forks after a new connection is accepted from a valid client and calls a special child() method that handles the connection between a client and mail server via POPFile for the life of the client connection. This is the beginning of a version of POPFile that supports a very large number of clients. (Requested by many people) 8. A new command line parameter -test_language can be set to 1 for UI translators. The UI will show the language identifiers used for each element rather than the strings associated with the identifier. 9. Rework of the History page to take into account ideas and suggestions from many people: Search now operates on From and Subject, From and Subject have popup windows to show the full text of each, you can sort by ID, From, Subject or Classification, you can filter by magnet classified mail, navigation is at the top and the bottom, state is remembered so that you can sort/filter and remove pages and reclassify and stay in the same place, many small improvements. 10. Major work on the MailParse.pm for dealing with improved mail parsing and the handling of spammers tricks (like S P A C E D out words etc.). Switch to using the Base64 and QuotedPrint modules from CPAN for improved mail cracking. (Work done mainly by sschinke) 11. Automated test suite now part of POPFile. (Suggested by aziegler) 12. Rework of all the HTML and CSS so that everything can be skinned and so that the complete UI is HTML 4.01 compliant. (Work done mainly by stanley_krute; there was a request from nobody for this) 13. You can now filter the History page by magnet filtered mails to look at them or delete them. (Requested by jarmstro, tcdk, msc_buff, fitzsimons, dyoungmcwicom) 14. The UI password no longer appears in URLs in plain text (requested by gribnif) 15. You can do multiple classifications at a time by selecting multiple messages and changing the drop down and hitting any Reclassify button. (Requested by dave823; implemented by sschinke) 16. You can now sort the History page by message ID, From, Subject and Classification (bucket). (Requested by jerometremblay, tomvoss, biobear) 17. When you remove a page the History no longer goes back to the start but stays at the same place in the History if it can. (Requested by ajweiner) 18. After you reclassify a message the History page stays at the same position (at the reclassified message). (Requested by harleythebrave, transit) 19. The installer now picks up old settings from a current POPFile install to fill in the options on the final dialog. (Requested by harleythebrave) 20. Added two new options -logdir and -msgdir to specify the location of the log directory (default to ./) and the messages (where the history is stored; defaults to messages/). (Requested by nobody) 21. Search in History now works on From as well as Subject. (Requested by transit) 22. The navigation from page to page in the History is now at the top and at the bottom of the page. (Requested by transit, nobody, nobody, nobody) 23. There are new [Remove] buttons on each message in the History to let you remove messages one by one. (Requested by nobody) 24. The X-POPFile-Link: repositions the History page at the message selected when you click the link. (Requested by zonk3r) 25. There is a new filter for unclassified messages. (Requested by nobody) 26. New -toptoo command line option enables classification on the POP3 TOP command and is only recommended for people on fast connections. (Requested by jberkes) 27. When viewing a colorized message there is a blank line between the header and the body of the message. (Requested by whiteg) 28. Message IDs (and History filenames) are now globally unique per POPFile install. (Requested by helphand) 29. You can now control the inclusion of the X-Text-Classification and X- POPFile-Link headers. (Requested by nobody) 30. Refreshing a page does not repeat the action any more. (Reported by nobody) 31. Fixed a bug where magnets with certain characters in them (e.g. [ or <) did not work and could not be deleted. (Reported by nobody, nobody, nobody, alanmuk, boneill, bana-sea-) 32. Fixed a bug where you could make POPFile exit with a divide by zero error if you started it with no corpus and then tried to Lookup a word's probability. (Reported by sunshine, gribnif) 33. Fixed a bug where the Shutdown link on the POPFile Control Center did not work if the UI password had been set. (Reported by actoews, nobody) 34. Fixed an embarassing bug where any POP server that contained the word 'stat' would not work with POPFile. (Reported by nobody, markabcxyz) 35. Fixed a bug where To magnets did not work correctly in the presence of other To style email headers. (Reported by dyoungmcwicom) 36. Fixed a bug where a specific email was causing the History page to lock up when you tried to view the colorized version. (Reported by ajweiner) 37. Fixed a bug where having pages in POPFile automatically refresh every ten minutes could cause POPFile to repeat an action (such as Reclassification) every 10 minutes. (Reported by mayfoev) 38. Fixed a bug where parts of a > attribute could appear in the From or Subject lines on the History (Reported and fixed by sschinke) 39. Fixed a bug where some CLS and MSG files were not getting deleted from the History when they were no longer needed. (Reported by helphand) TRANSLATIONS Thanks to the following for helping to make POPFile a global piece of software: Brazilian Adriano RG Danish Nicki Brøchner Nielsen Dutch Jilles Oldenbeuving French Antony Boucher German Matthias Deege Norweigian Jo Inge Fjellstad Russian Alexander Saltanov Please translate the UI and the manual into your language. DONATIONS Thank you to everyone who as clicked the Feed Me! button and donated their hard earned cash to me in support of POPFile. Thank you also to the people who have contributed patches, feature requests and bug reports. Keep the ideas and bug reports coming. John.