Bleeding Edge - Source Code → TODOs for v1.1.4 and CHANGES from v1.1.3

TODOs for v1.1.4 and CHANGES from v1.1.3

Hi folks,

I come back after a long interval. Sorry for long absence.

Now I think it's time to release another minor update (v1.1.4).

TODOs for v1.1.4:

I'll look into these problems.

  • Enhancement
    • Brian is working to update Windows installer
    • Joseph is working to update skins

CHANGES from v1.1.3:

  • Bug fixes
    • Fixed a bug that in some cases clicking "Reclassify" without selecting bucket name would result in database corruption. 3817 ticket#171 (Brian)
  • Minor bug fixes
    • Fixed a problem that POPFile takes a long time to process messages which has a very long 'To' header. 3821
    • Fixed a bug that causes an error when another POPFile process is running. 3836 ticket#179
    • Log messages and error messages will be shown in correct order on Windows. 3837
    • Avoid SSL_verify_mode warning. 3844

Is there any other work to do?

Naoki

  • Message #1779

    Some notes on the Windows Installer for POPFile 1.1.4

    The Windows version of POPFile 1.1.3 was built using version 2.45 of the NSIS compiler and Perl 5.8.9.

    NSIS 3.0 will be the first official Unicode version of NSIS. Although NSIS 3.0a1 is available it is an alpha release. I intend to use NSIS 2.46 (the most recent ANSI version) for POPFile 1.1.4 because it will make it easier for me to update the Windows installer :)

    (NSIS 3.x will be used for POPFile 2.x because a lot of changes are needed for the Windows version of POPFile 2.x)

    The Windows installer uses some extra NSIS plugins and several of these have been updated since POPFile 1.1.3 was released. I'll need to make sure the installer works with the updated plugins.

    A newer version of Perl will be used for POPFile 1.1.4, probably either Perl 5.14 or Perl 5.16 because these are the versions freely available from ActiveState's PPM system. Now that Perl 5.18 has been released, support for Perl 5.14 may come to an end "soon" so perhaps Perl 5.16 should be chosen instead of Perl 5.14?

    Earlier versions of Perl require an expensive licence and Perl 5.18 support is not yet listed in the PPM index at http://code.activestate.com/ppm/

    A new release of POPFile has always been able to upgrade the database used by any previous release. In order to do this the Windows version of POPFile 1.1.3 includes support for BerkeleyDB and SQLite 2.x format databases. This allows POPFile to automatically convert these old databases into the current SQLite 3.x format. I've not installed Perl 5.14 or 5.16 yet so I don't know if support for these old database formats can be easily provided in the Windows installer for POPFile 1.1.4.

    Brian

    • Message #1780

      Brian,

      I think using Perl 5.16 is better than 5.14.
      I've created a new page on wiki: http://getpopfile.org/docs/devel:activeperlppms

      DBD::SQLite2, Text::Kakasi, IO::Socket::SSL and Net::SSLeay modules seem to be unavailable on ActiveState? repository.

      BerkeleyDB was only used between v0.20.0 (released on October 17,2003. Ten years ago!) to v0.20.1 (released on November 4, 2003) and POPFile v0.21.0 (released on March 9, 2004) uses SQLite 2.x.
      I think we can drop the BerkeleyDB support.

      Naoki

      • Message #1781

        Naoki,

        I think using Perl 5.16 is better than 5.14. I've created a new page on wiki

        OK, I'll try using Perl 5.16 for POPFile 1.1.4. Thanks for the new wiki page.

        DBD::SQLite2, Text::Kakasi, IO::Socket::SSL and Net::SSLeay modules seem to be unavailable

        ActiveState used to provide DBD::SQLite2 for Perl 5.8 and that is what I've been using up until now. Their PPM no longer provides any Perl 5.8 modules without an expensive licence.

        I've been using Text::Kakasi from the University of Winnipeg repository, IO::Socket::SSL from tcool (Kenichi Ishigaki's repository) as that has the extra features needed for your code and Net::SSLeay from bribes (the Bribes de Perl repository) because it offers a more up-to-date version than the tcool or University of Winnipeg repositories.

        The wiki has some notes on what I've been using for the Windows installer for previous releases: http://getpopfile.org/docs/devel:windowsinstaller

        BerkeleyDB was only used [for POPFile releases between] ... October 17,2003 ... and ... March 9, 2004 ... I think we can drop the BerkeleyDB support.

        I had forgotten how long it had been since POPFile used BerkeleyDB. I will have to spend some time on the new minimal Perl so I may as well try the current BerkeleyDB package just to see if it works with an old POPFile database.

        Brian

        • Message #1783

          Brian,

          ActiveState used to provide DBD::SQLite2 for Perl 5.8 and that is what I've been using up until now. Their PPM no longer provides any Perl 5.8 modules without an expensive licence.

          I've been using Text::Kakasi from the University of Winnipeg repository, IO::Socket::SSL from tcool (Kenichi Ishigaki's repository) as that has the extra features needed for your code and Net::SSLeay from bribes (the Bribes de Perl repository) because it offers a more up-to-date version than the tcool or University of Winnipeg repositories.

          I've installed Perl 5.16 (x86) on Windows 8 Pro and found IO::Socket::SSL (1.84) and Net::SSLeay (1.52) are installed by default.
          Bribes Perl repository has more up-to-date version of these modules.

          As for DBD::SQLite2, currently I can't find any binary modules.

          Akaneiro provides PPD files of Text::Kakasi for ActivePerl 5.8.x (x86), 5.14.x (x64) and 5.16.x (x64): http://www.akaneiro.jp/PPMPackages/
          The binary for 5.16.x is only available in 64bit version, so I've built x86 (32bit) version of Text::Kakasi module: http://amatubu.skr.jp/ppm/Text-Kakasi.ppd
          You can install it using ppm command:

          > ppm install http://amatubu.skr.jp/ppm/Text-Kakasi.ppd
          

          I'm testing POPFile in this environment:

          System : MSWin32
          Perl : 5.016003
          
          Installed Perl modules
          
             BerkeleyDB                   <not installed>
             Carp                         1.260
             Cwd                          3.400
             DBD::SQLite                  1.370
             DBI                          1.623
             Date::Format                 2.240
             Date::Parse                  2.300
             Digest::MD5                  2.520
             Encode                       2.550
             Encode::Guess                2.060
             Fcntl                        1.110
             File::Copy                   2.230
             File::Glob::Windows          0.100
             Getopt::Long                 2.380
             HTML::Tagset                 3.200
             HTML::Template               2.940
             IO::Handle                   1.330
             IO::Select                   1.210
             IO::Socket                   1.340
             IO::Socket::INET             1.330
             IO::Socket::SSL              1.954
             IO::Socket::Socks            0.620
             MIME::Base64                 3.130
             MIME::QuotedPrint            3.130
             MeCab                        <not installed>
             POSIX                        1.300
             Sys::Hostname                1.160
             Text::Kakasi                 2.040
             XMLRPC::Transport::HTTP      0.717
          

          The wiki has some notes on what I've been using for the Windows installer for previous releases: http://getpopfile.org/docs/devel:windowsinstaller

          Thanks. That page is very useful!

          Naoki

          • Message #1796

            Naoki,

            The wiki has some notes on what I've been using for the Windows installer for previous releases: http://getpopfile.org/docs/devel:windowsinstaller


            Thanks. That page is very useful!

            I am glad you found it useful. After my hard disk failed a few years ago I had a lot of trouble setting things up to build the installer on my new disk; there are so many bits and pieces needed!

            I decided to build POPFile 1.1.4's minimal Perl from scratch, instead of trying to modify the one from 1.1.3. There are more changes than I expected. Perl 5.16 seems quite different from Perl 5.8.

            Today I am going to start on the extra Perl requirements for Kakasi. It has been a long time since I last looked at the Nihongo parsers. I think all I need to do is select 'Nihongo' as the UI language and then feed some Nihongo test messages to POPFile ... and then respond to the Perl error messages.

            Brian

            • Message #1801

              I decided to build POPFile 1.1.4's minimal Perl from scratch, instead of trying to modify the one from 1.1.3. There are more changes than I expected. Perl 5.16 seems quite different from Perl 5.8.

              Do not pay much attention to my recent SVN activity. I think I made some bad decisions when preparing the new minimal Perl. I am going to try again to see if I can make a better minimal Perl.

              I think the popfile*.exe files are OK because I did not change anything, I just rebuilt them to use Perl 5.16.

              Brian

  • Message #1812

    TODOs for v1.1.4:
    * Problems should be solved (Is there any other problems to solve?)
    * TLS issue (POPFile does not support TLS) http://getpopfile.org/discussion/1/463

    I think I've found the cause of the problem and created a new ticket:

    http://getpopfile.org/ticket/193

    I think we can solve the problem by updating SSL modules (IO::Socket::SSL and Net::SSLeay).

    Naoki

    • Message #1822

      I think we can solve the problem by updating SSL modules (IO::Socket::SSL and Net::SSLeay).

      Net::SSLeay is supposed to be thread-safe now so we should be able to allow concurrent POP3 connections when using SSL connections to mail servers. The Net::SSLeay documentation says:

      Using Net::SSLeay in multi-threaded applications

      IMPORTANT: versions 1.42 or earlier are not thread-safe!

      Net::SSLeay module implements all necessary stuff to be ready for multi-threaded environment - it requires openssl-0.9.7 or newer. The implementation fully follows thread safety related requirements of openssl library(see http://www.openssl.org/docs/crypto/threads.html).

      If you are about to use Net::SSLeay (or any other module based on Net::SSLeay) in multi-threaded perl application it is recommended to follow this best-practice:

      This is followed by sections covering 'Initialization', 'Using callbacks', 'Using openssl elements', 'Using other perl modules based on Net::SSLeay' and 'Combining Net::SSLeay with other modules linked with openssl'

      Brian

      (Edited to fix the garbled link in the quote from the Net::SSLeay documentation)

      • Message #1831

        Net::SSLeay is supposed to be thread-safe now so we should be able to allow concurrent POP3 connections when using SSL connections to mail servers.

        That is a good news!

        POPFile v1.1.3 includes Net::SSLeay v1.55, so we can test whether we can allow concurrent POP3 connections with newer version of Net::SSLeay!

        Naoki

        • Message #1832

          POPFile v1.1.3 includes Net::SSLeay v1.55, so we can test whether we can allow concurrent POP3 connections with newer version of Net::SSLeay!

          Oops, I'm sorry. I had updated IO::Socket::SSL and Net::SSLeay manually to test the TLS problem.

          I'll test my modified environment.

          Naoki

          • Message #1833

            I'll test my modified environment.

            OK, the 'concurrent POP3 connection' works with SSL on Windows!

            Here's the log:

            2014/1/24 22:16:07 -2612: pop3: 213: Command: --USER XXXXXX--
            2014/1/24 22:16:07 -2612: pop3: 529: Attempting to connect to SSL server at pop.gmail.com:995
            <snip>
            2014/1/24 22:16:09 -2612: pop3: 558: Connected to pop.gmail.com:995 timeout 60
            2014/1/24 22:16:09 -2612: pop3: 606: Connection returned: +OK Gpop ready for requests from 61.125.146.242 h4pf18307780pat.28[0d][0a]
            <snip>
            2014/1/24 22:16:10 -2612: pop3: 311: USER XXXXXX
            2014/1/24 22:16:10 -2612: pop3: 659: Read slurp data +OK send PASS XXXXXX
            2014/1/24 22:16:10 -2612: pop3: 311: +OK send PASS XXXXXX
            2014/1/24 22:17:00 -2612: pop3: 213: Command: --PASS XXXXXX--
            2014/1/24 22:17:00 -2612: pop3: 311: PASS XXXXXX
            2014/1/24 22:17:08 -2612: pop3: 659: Read slurp data +OK Welcome.[0d][0a]
            2014/1/24 22:17:08 -2612: pop3: 311: +OK Welcome.[0d][0a]
            2014/1/24 22:17:09 3064: mq: 227: Parent: postfork() called for pid -2660, reader GLOB(0x1f5b950)
            <snip>
            2014/1/24 22:19:54 -2612: pop3: 213: Command: --RETR 1--
            2014/1/24 22:19:54 -2612: pop3: 311: RETR 1[0d][0a]
            2014/1/24 22:19:55 -2612: pop3: 659: Read slurp data +OK message follows[0d][0a]
            2014/1/24 22:19:55 -2612: pop3: 311: +OK message follows[0d][0a]
            2014/1/24 22:19:55 -2612: history: 300: reserve_slot selected random number 1251222
            2014/1/24 22:19:55 -2612: history: 318: reserve_slot returning slot id 1
            <snip>
            2014/1/24 22:24:04 -5992: pop3: 213: Command: --USER XXXXXX--
            2014/1/24 22:24:04 -5992: pop3: 529: Attempting to connect to SSL server at amatubu.sakura.ne.jp:995
            2014/1/24 22:24:05 -5992: pop3: 558: Connected to amatubu.sakura.ne.jp:995 timeout 60
            2014/1/24 22:24:05 -5992: pop3: 606: Connection returned: +OK Hello there. <10387.1390569836@www229.sakura.ne.jp>[0d][0a]
            <snip>
            2014/1/24 22:24:05 -5992: pop3: 311: USER XXXXXX
            2014/1/24 22:24:05 -5992: pop3: 659: Read slurp data +OK Password required.[0d][0a]
            2014/1/24 22:24:05 -5992: pop3: 311: +OK Password required.[0d][0a]
            2014/1/24 22:24:13 -5992: pop3: 213: Command: --PASS XXXXXX--
            2014/1/24 22:24:13 -5992: pop3: 311: PASS XXXXXX
            2014/1/24 22:24:15 -5992: pop3: 659: Read slurp data +OK logged in.[0d][0a]
            2014/1/24 22:24:15 -5992: pop3: 311: +OK logged in.[0d][0a]
            <snip>
            2014/1/24 22:24:19 -5992: pop3: 213: Command: --RETR 1--
            2014/1/24 22:24:19 -5992: pop3: 311: RETR 1[0d][0a]
            2014/1/24 22:24:19 -5992: pop3: 659: Read slurp data +OK 1189 octets follow.[0d][0a]
            2014/1/24 22:24:19 -5992: pop3: 311: +OK 1189 octets follow.[0d][0a]
            2014/1/24 22:24:19 -5992: history: 300: reserve_slot selected random number 1251222
            2014/1/24 22:24:19 -5992: history: 318: reserve_slot returning slot id 3
            <snip>
            2014/1/24 22:24:23 -2612: pop3: 213: Command: --RETR 3--
            2014/1/24 22:24:23 -2612: pop3: 311: RETR 3[0d][0a]
            2014/1/24 22:24:24 -2612: pop3: 659: Read slurp data +OK message follows[0d][0a]
            2014/1/24 22:24:24 -2612: pop3: 311: +OK message follows[0d][0a]
            2014/1/24 22:24:24 -2612: history: 300: reserve_slot selected random number 193298341
            2014/1/24 22:24:24 -2612: history: 318: reserve_slot returning slot id 4
            

            I'll commit the changes to the repository.

            Naoki

  • Message #1856

    TODOs for v1.1.4:
    * Problems should be solved (Is there any other problems to solve?)
    * TLS issue (POPFile does not support TLS) http://getpopfile.org/discussion/1/463

    This problem will be solved by updating SSL modules.

    * flush file issue (in some situation flush files are not deleted) http://getpopfile.org/discussion/1/465 ticket#188

    I've updated the way to store contents of e-mails. POPFile will never create any flush files.

    * long time to load UI issue (Windows IE only?) http://getpopfile.org/discussion/1/408

    This seems to be a problem of the Web browser (Windows IE). Currently I have no clue to solve the issue.

    Now, here's a draft of release notes for v1.1.4:

    CHANGES from v1.1.3:

    • Enhancements
      • Windows version now uses Perl 5.16.x (TODO)
      • Skins are updated? (TODO)
      • SSL modules included in Windows version are updated and now users can use POP3 concurrent connections with SSL on Windows. (3858 and 3869)
      • POPFile can now verifies peer (server) certificates when users turn on a new option 'GLOBAL_ssl_verify_peer_certs'. (3863)
    • Bug fixes
      • Fixed a bug that in some cases clicking "Reclassify" without selecting bucket name would result in database corruption. (3817 and ticket#171) (Brian)
      • Fixed a bug that POPFile can't connect to some servers via SSL. (ticket#193)
      • Fixed a bug that in some situation flush files are not deleted. (3849)
    • Minor bug fixes
      • Fixed a problem that POPFile takes a long time to process messages which has a very long 'To' header. (3821)
      • Fixed a bug that causes an error when another POPFile process is running. (3836 and ticket#179)
      • Log messages and error messages will be shown in correct order on Windows. (3837)

    Naoki

    • Message #1946

      How is the update coming along?

      • Message #1947

        Progress on the Windows version has been rather slow for a number of reasons.

        The minimal Perl is being completely replaced and will therefore require more testing than usual (e.g. the dependencies in Perl 5.16.x are very different from those in Perl 5.8.x).

        A beta version has just been built to let me test the Windows version of POPFile 1.1.4.

        Further changes to the Windows installer need to be made before a Release Candidate can be produced.

        • Message #2137

          I don't know much about using SSL with POPFile, but that error message mentions TLS which is a type of security. From the wiki, it looks like TLS is not supported yet.
          Thanks
          http://www.fixithere.net/sky-customer-service/