Help → Upgrade is a degrade

Upgrade is a degrade

I've been running POPFile for several years now - possibly since v0.18 or earlier and up until now it has worked perfectly, but had not felt the need to upgrade to the more recent builds [If it ain't broke ...].

With a new machine build just completed over the weekend I figured it might be good to install the latest version: what I got is something that - as far as my mail client is concerned - doesn't work at all.

The old version of POPFile was configured to do both header injection thus ...

X-Text-Classification: spam
X-POPFile-Link: http://127.0.0.1:8080/jump_to_message?view=253583

and to modify the subject line with the string " [spam] " at the start of the subject line.

The "new" version is set to do the same - except that it doesn't do anything of the kind. No subject modification and no X-Text-Classification or X-POPFile-Link headers.

AntiVirus? [Avast] configuration is identical to the previous version.

Since I still have my old machine I am able to compare the old and new configurations: they're effectively identical; same magnets, same buckets etc etc.

I've looked at every setting in the UI, and all the configuration files I can find and can see no reason why this should be happening. If anyone has a clue I'd be very grateful for some pointers.

RN

  • Message #1248

    Maybe look at the toptoo setting maybe the email cleint is doing a two pass mail gatehr so it nevers see the added text. If you look at the Popfile web list of the emails do they have the data you're looking for but your email client doesn't see it or is it not in the popfile detail of the email either?

    bk

    • Message #1249

      Thanks for the thought but no; that is not it.

      Settings on the old machine are identical to those on the new.
      Old machine with old version of POPFile works, new machine with current version of POPFile does not. There are no POPFile X Headers visible in POPFile Web UI or in the mail client.

      • Message #1250

        It would help if you mentioned which operating system you are using and the two versions of POPFile (on old machine and on new machine).

        There are no POPFile X Headers visible in POPFile Web UI

        What do you mean by this?

        • Message #1251

          Both machines: XP [x86] SP3 with all current hotfixes etc.

          PopFile? versions:
          Old Machine = 0.22.4
          New Machine = 1.1.1

          Other data: AV = Avast 4.8. running on default ports - thus POP port = 110, with POPFile also running on port 110. Avast does NOT have redirection turned on and is not listening on alternate ports. Popfile is talking directly to Avast.

          Telnet to localhost port 110 works as expected, with a welcome message from POPFile - I can then log in to the remote mail server manually: when downloading messages via telnet I can "see" Avast processing the stream - and this is demonstrated by the presence of Avast X-Headers in the messages after the download has completed.

          >There are no POPFile X Headers visible in POPFile Web UI
          What do you mean by this?

          Exactly what it says: open any message on the new machine from the web UI and there are no X-Popfile headers of any kind - even though PopFile? is set [by default] to add them for all categories [buckets]. There is also no evidence in the message - as received by the mail client - that POPFile has done anything at all to the message after classifying the item: there are no X-Popfile headers visible there either.

          Popfile is classifying messages correctly, as can be seen in the History page, but it's not writing headers. Which means that several year's worth of carefully developed mail rules are failing.

          • Message #1252

            There are no POPFile X Headers visible in POPFile Web UI
            What do you mean by this?


            Exactly what it says: open any message on the new machine from the web UI and there are no X-Popfile headers of any kind

            Sorry, I still don't understand what you mean. The Single Message View in the POPFile UI does not show any X-Text-Classification or X-POPFile-Link headers. I think the BUCKETS page is the only POPFile UI page that mentions these special POPFile headers.

            • Message #1253

              You're quite correct about X-Headers only being visible/mentioned on the Buckets page.

              In my malfunctioning copy of POPFile those headers are 'set' ON for all buckets - which as far as I can tell is the default install condition. POPFile is classifying the messages correctly as seen in the History pages - and SHOULD BE adding appropriate headers and making subject modifications, but it isn't doing either of those things.
              As far as the email client is concerned, POPFile is doing nothing at all.

              As for the raw message display: Go to POPFile UI/History.

              Click on any message in the Subject column.

              What is then displayed is a] how POPFile currently categorises the message and b] the entire message in "raw" format: complete with all received headers.

              I failed to write clearly in my original message: what this page SHOULD be showing - as well as the received raw message - is the "logical" or resultant set of policy consequence of all the rules POPFile has so far applied to that message - where the top of the page shows the message categorised as spam/personal/work/other [with option to reclassify] it SHOULD ALSO show which header or other rules have been applied to that message, and possibly also show stuff like any "new" dictionary words or stats. It's displaying the end point of a process - which is more than simply the gross bucket classification.

              That means it should allow users to see [or work out] what POPFile actually sent to the email client: whether or not the subject was modified [and how], whether a X-POPFile-Link header was inserted, and so on. Comparing such output with what the email client actually received would help with process diagnosis.

              There is nowhere in POPFile UI where I can track down what the application is actually doing to inbound messages on a case by case basis - I can see the _results_ of its actions in my email client, but there's nowhere I can follow the individual PF processes through to see where it is [or might be] failing.

              I should add that even though I've set logging level to max [2], no logs are actually being written. Clearly something is broken, but it's not clear to me what that is.

              • Message #1255

                As for the raw message display: Go to POPFile UI/History.

                Click on any message in the Subject column.

                What is then displayed is a] how POPFile currently categorises the message and b] the entire message in "raw" format: complete with all received headers.

                This is the "Single Message View" which I referred to (and it is identified as such near the top of the UI page).

                Clearly something is broken, but it's not clear to me what that is.

                Since you are running Windows XP you can use some of the diagnostic software supplied with POPFile:

                (1) Run the detailed (i.e. full) diagnostic utility:

                Start -- All Programs -- POPFile -- Support -- PFI Diagnostic utility (full)

                (2) Shut down POPFile then check its database:

                Start -- All Programs -- POPFile -- Support -- Check database status

                Both utilities display a report in a scrollable window and if you right-click inside the window you can copy the report to the clipboard and paste it into a forum reply.

                AV = Avast 4.8. running on default ports - thus POP port = 110, with POPFile also running on port 110.

                I suggest you change POPFile's POP3 listening port to 123 and change your email client to use port 123 for the accounts which are being handled by POPFile. This will make things simpler.

                Brian

                • Message #1256

                  Thanks Brian,

                  I had already run both diagnostics before first posting here; there's nothing obvious in any of the output which indicates any problems to me. I've just re-run them at your suggestion and the DB report is entirely clean: the only line in the full PF report that _might_ indicate to me that there's a possible issue is

                  "HKLM: NewParser? = ><"

                  The full log is at the foot of this message.

                  As for switching the PopFile/Avast? ports:

                  1] that suggestion directly contradicts the current documentation here for use of PF with Avast versions >=Avast 4.6

                  2] I had already tried switching the ports and it made no difference to the behaviour of POPFile.
                  I had also tried a] with Avast mail scanning turned off and mail client and POPFile using port 110 and b] Avast mail scanning off and using port 123 for PF and mail client. The results are the same in all cases.

                  Regardless of any port settings in PF or the mail client, all email received at the client has no PF X-Headers or changes to the subject line: which is why I changed the ports back to the default setting and turned AV scanning on again.
                  And yes, I did restart POPFile and the mail client - and the machine - after each set of changes.

                  3] from a purely technical perspective when talking about mail client to PF relationships, switching I/O ports should only make an obvious and noticeable difference to behaviour when there's a NAT or NPT or security process [such as SOCKS or SPI] between mail client and POPFile, or possibly between POPFile and the mail server.
                  In all other circumstances which I can think of, if the email client and POPFile are properly designed and implemented - and working as designed, then the client relationship with POPFile should just work, even when they're using the same port assignations - and even if there's an AV process running on the same port.
                  Which is exactly what happens on my "old" machine using the older version of POPFile - it simply works - and has done faultlessly for several years - all running on port 110.

                  With all that in mind I'd welcome clarification of what you mean by "make things simpler" - to my mind switching ports makes things more complex, not less.

                  The issue here isn't whether or not POPFile is successfully receiving and classifying email; it clearly IS doing that - and doing it AFTER Avast has processed the inbound emails. What POPFile is failing to do is write the X-headers and prepend a tag to the subject line before "forwarding" to the email client - and it still fails to do that even when there is no AV process in the mail chain.

                  I'm rather afraid that unless you can convince me otherwise I will continue to believe that this isn't an issue of email client configuration, or of AV configuration; it's either a design or a process and configuration issue in the current version of POPFile. I will however freely admit that I'm a technical user who may know just enough to be dangerous!

                  I'm _almost_ tempted to uninstall POPFile, clean out the registry and DB etc and then reinstall with default settings, just to see if the behaviour changes.

                  Let me know if there's any further diagnostics you'd like me to carry out.


                  POPFile PFI Diagnostic Utility v0.1.14 (full mode)


                  String data report format (not used for numeric data)

                  string not found : ><
                  empty string found : < >
                  string with 'xyz' value found : < xyz >


                  Current UserName? = Robert (Admin)

                  Windows version = XP Professional
                  IsNT return code = 1
                  Internet Explorer = 7.0


                  Location used to store temporary files


                  $TEMP folder path = < C:\DOCUME~1\Robert\LOCALS~1\Temp >


                  Start Menu Locations


                  AU: $SMPROGRAMS = < C:\Documents and Settings\All Users\Start Menu\Programs >
                  AU: $SMSTARTUP = < C:\Documents and Settings\All Users\Start Menu\Programs\Startup >

                  Search results for the "AU: $SMSTARTUP" folder:

                  *.lnk files found = 2
                  POPFile shortcuts = 0

                  CU: $SMPROGRAMS = < C:\Documents and Settings\Robert\Start Menu\Programs >
                  CU: $SMSTARTUP = < C:\Documents and Settings\Robert\Start Menu\Programs\Startup >

                  Search results for the "CU: $SMSTARTUP" folder:

                  Shortcut name = < Run POPFile.lnk >
                  Shortcut start in = < C:\Program Files\POPFile >
                  Shortcut target = < C:\Program Files\POPFile\runpopfile.exe >
                  Shortcut argument = < /startup >
                  Target status = found

                  *.lnk files found = 3
                  POPFile shortcuts = 1


                  Obsolete/testbed Registry Entries


                  [1] Pre-0.21 Data:

                  Pre-0.21 POPFile = ><
                  Pre-0.21 Testbed = ><

                  [2] 0.21 Test Installer Data:

                  HKLM: RootDir?_LFN = ><
                  HKLM: RootDir?_SFN = ><

                  HKCU: RootDir?_LFN = ><
                  HKCU: RootDir?_SFN = ><
                  HKCU: UserDir?_LFN = ><
                  HKCU: UserDir?_SFN = ><

                  [3] Current PFI Testbed Data:

                  MRI PFI Testbed = ><
                  MRI PFI Testdata = ><


                  POPFile Registry Data


                  NTFS SFN Disabled = < 0 >

                  HKLM: MRI Version = < 1.1.1 >

                  HKLM: NewParser? = ><

                  HKLM: InstallPath? = < C:\Program Files\POPFile >
                  HKLM: RootDir?_LFN = < C:\Program Files\POPFile >
                  HKLM: RootDir?_SFN = < C:\PROGRA~1\POPFile >

                  HKLM: *.exe count = 6 (this is OK)

                  HKCU: Data Owner = < Robert >
                  HKCU: MRI Version = < 1.1.1 >
                  HKCU: RootDir?_LFN = < C:\Program Files\POPFile >
                  HKCU: RootDir?_SFN = < C:\PROGRA~1\POPFile >

                  HKCU: UserDir?_LFN = < C:\Documents and Settings\Robert\Application Data\POPFile >
                  HKCU: UserDir?_SFN = < C:\DOCUME~1\Robert\APPLIC~1\POPFile >

                  HKCU: popfile.pl = found
                  HKCU: popfile.cfg = found

                  HKCU: *.exe count = 6 (this is OK)


                  POPFile Environment Variables


                  'POPFILE_ROOT' = < C:\PROGRA~1\POPFile >
                  'POPFILE_USER' = < C:\DOCUME~1\Robert\APPLIC~1\POPFile >

                  Env: popfile.pl = found
                  Env: popfile.cfg = found

                  ROOT: *.exe count = 6 (this is OK)

                  'ITAIJIDICTPATH' = >< (this is OK)
                  'KANWADICTPATH' = >< (this is OK)

                  'MECABRC' = >< (this is OK)


                  (report created 17-Mar-2010 @ 00:30:16)


                  • Message #1257

                    I'll let Brian comment on those test results, but something is obviously wrong with your installation. The fact that you aren't able to get any logging from POPFile is highly suspicious. I have no idea what kind of configuration error could lead to empty logfiles and POPFile neglecting to modify the email headers, but I think both problems are likely to have the same cause.

                    Is the logfile created and empty or is it simply not there? All you have running on your new machine is the new version of POPFile and you didn't upgrade the old install but simply installed the new version?

                    Hope we can sort this out soon.

                    Regards,
                    Manni

                    • Message #1259

                      Manni

                      I agree with you about the circs being suspicious; it's why I'm here asking questions :-)

                      No logs created; No prior version of PF installed on the "new" machine.

                      Robert

                  • Message #1258

                    the DB report is entirely clean

                    I'd still like to see it (that is why I asked you to run it).

                    "HKLM: NewParser? = ><"

                    You can safely ignore this; it only applies when you are using POPFile's Japanese/Nihongo mode.

                    As for switching the PopFile/Avast? ports:

                    1] that suggestion directly contradicts the current documentation here for use of PF with Avast versions >=Avast 4.6

                    If you are referring to Latest Version of AVAST (4.5) that page has not been updated for over three years and appears to say that the current Avast version is 4.5. Avast 5.0 was released earlier this year. Why are you still using an old version of Avast? I thought version 5.0 was supposed to be a lot better than 4.8.

                    2] I had already tried switching the ports and it made no difference to the behaviour of POPFile.

                    With all that in mind I'd welcome clarification of what you mean by "make things simpler" - to my mind switching ports makes things more complex, not less.

                    Your anti-virus program is probably running a transparent POP3 proxy on port 110. When POPFile's POP3 listening port is set to 110 then it is also running a transparent proxy on port 110. So which path does your POP3 mail take?

                    email client -- port 110 -- POPFile -- port 110 -- Avast -- internet
                    email client -- port 110 -- Avast --- port 110 -- POPFile -- internet
                    email client -- port 110 -- Avast --- port 110 -- POPFile --- port 110 -- Avast -- internet
                    or some other sequence?

                    By changing POPFile's POP3 listening port to 123 the proxy chain is simpler:

                    email client -- port 123 -- POPFile -- port 110 -- Avast -- internet

                    I still suggest you make this change. It makes the POP3 proxy chain simpler and hopefully more robust. I did not claim you'd "see" any difference; the improvements all take place "under the hood". Several users have found that making this change helps. I have been using this scheme for many years with several different anti-virus packages. Of course you can ignore my advice.

                    I should add that even though I've set logging level to max [2], no logs are actually being written.

                    What are the values of the three "logger" entries in the ADVANCED page of the UI?

                    I'm _almost_ tempted to uninstall POPFile, clean out the registry and DB etc and then reinstall with default settings, just to see if the behaviour changes.

                    How did you install POPFile on the new machine?

                    How did you transfer your POPFile settings and database from the old machine to the new machine?

                    Are there lots of files in your temoprary files folder?

                    How much free disk space is there?

                    By default POPFile console messages are hidden. Shut down POPFile and then use the Message Capture utility (the utility will run POPFile for you):

                    Start -- All Programs -- POPFile -- Support -- Message Capture utility

                    The utility will display some messages in a scrollable window. POPFile will be ready to handle email after the window shows the "POPFile Engine v1.1.1 running" message.

                    Try checking for email. Some warning or error messages may appear in this window. You can copy and paste the report into a reply here. In addition to possible warnings and errors the report contains information about POPFile's configuration so I'd like to see the report whatever happens.

                    Brian

                    • Message #1260

                      Brian

                      Sorry: I did say I was a user who knew a dangerous and possibly misleading bit about IT. The DB report was clean so I assumed it would not really be required: here it is


                      POPFile SQLite Database Status Check (integrated) v0.2.0


                      Current user : Robert
                      Current folder: C:\Program Files\POPFile
                      Command line : /REGISTRY

                      Trying to find database using registry data (HKCU)... found it!

                      POPFile database found (C:\Documents and Settings\Robert\Application Data\POPFile\popfile.db)

                      SQLite v3.6.13 utility found in C:\Program Files\POPFile

                      Database is in SQLite 3.x format, uses schema version 3 and its size is 1,123 KB

                      Result of running the 'pragma integrity_check;' command:
                      ok

                      The POPFile database has passed the SQLite integrity check!


                      (report finished 17-Mar-2010 @ 16:28:06)


                      and as you can see, it seems to be showing no errors or problems.

                      You can safely ignore this; it only applies when you are using POPFile's Japanese/Nihongo mode.

                      Yup: that's what I'd figured out.

                      1] that suggestion directly contradicts the current documentation here for use of PF with Avast versions >=Avast 4.6


                      If you are referring to Latest Version of AVAST (4.5) that page has not been updated for over three years and appears to say that the current Avast version is 4.5. Avast 5.0 was released earlier this year. Why are you still using an old version of Avast? I thought version 5.0 was supposed to be a lot better than 4.8.

                      I'll deal with that in two parts:

                      a] the underlying architecture of Avast hasn't changed since the semi-major revision at version 4.5 - so Avast's behaviour hasn't changed even with the release of v5. All that /has/ changed are the detection engines: the scanning and port management process is identical.

                      b] because I have a site licence for multiple machines and I am maintaining consistency. Better is also a very subjective term - version 5 may arguably be better, but it's also got a bigger disk/ram footprint and is marginally slower when running on the same platform as 4.8. Since the signature DB for 4.8 is the same as for 5 there's no significant loss of security by not upgrading at this time. We'll upgrade when the current site licence expires in about 9-10 weeks.

                      Your anti-virus program is probably running a transparent POP3 proxy on port 110. When POPFile's POP3 listening port is set to 110 then it is also running a transparent proxy on port 110. So which path does your POP3 mail take?

                      Yes, it is running a transparent proxy.

                      The configuration is

                      email client -- port 110 -- POPFile -- port 110 -- Avast -- internet

                      By changing POPFile's POP3 listening port to 123 the proxy chain is simpler:

                      We disagree.

                      1] It's worked perfectly well in this configuration on my old machine using port 110 throughout for many years.

                      2] over the last 25 years I've acquired more than a little experience of designing developing deploying and managing hundreds of thousands of mail clients for end users, used with and without proxies; I've never needed to change the mail client port{s} as you suggest _except_ when working through inline SOCKS or IAS proxies or routers that needed special configs to work with multiple users/machines or multiple public IPs etc.

                      This is all about the correct use of "sockaddr" code and the binding order of the port. If a process such as an AV mail scanner binds higher up the order then PF then it will appear entirely transparently to PF - as if it is the NIC. If an AV process is installed before PF then it will bind first, and PF should have no need [nor should it try] to alter that - and should work perfectly transparently with the config it encounters on installation.

                      Like I said, I'm a user with some - possibly dangerous - knowledge. So broadly speaking, you haven't yet convinced me that there's a sound reason or need to switch the ports, but even setting that issues aside, ports are NOT the problem here: if they were then removing Avast from the mail collection chain should allow PF to work correctly. It doesn't.

                      I should add that even though I've set logging level to max [2], no logs are actually being written.


                      What are the values of the three "logger" entries in the ADVANCED page of the UI?

                      logger_format default
                      logger_level 2
                      logger_logdir /

                      How did you install POPFile on the new machine?

                      It was a clean machine; installed the Avast AV, "installed" mail client by making a shortcut from the server client process [it's not Outlook or TBird or any other common client], installed POPFile by running setup.exe.

                      How did you transfer your POPFile settings and database from the old machine to the new machine?

                      I didn't.

                      Are there lots of files in your temoprary files folder?

                      Which temp folder? System temp or User temp?
                      System temp is virtually empty, user temp has about 40-50 files totalling some 25Mb or so.

                      How much free disk space is there?

                      In excess of 100Gb.

                      Try checking for email. Some warning or error messages may appear in this


                      POPFile Message Capture Utility v0.1.15


                      POPFILE_ROOT = C:\PROGRA~1\POPFile
                      POPFILE_USER = C:\DOCUME~1\Robert\APPLIC~1\POPFile
                      Using 'popfileif.exe' to run POPFile


                      (report started 17-Mar-2010 @ 17:00:31)


                      POPFile Engine loading

                      Loading...

                      {core: windows}
                      {core: config history logger mq}
                      {classifier: bayes wordmangle}
                      {interface: html xmlrpc}
                      {proxy: nntp pop3 smtp}
                      {services: imap}

                      POPFile Engine v1.1.1 starting

                      Initializing...

                      {core: config history logger mq windows}
                      {classifier: bayes wordmangle}
                      {interface: html xmlrpc}
                      {proxy: nntp pop3 smtp}
                      {services: imap}

                      Starting...

                      {core: config history logger mq windows}
                      {classifier: bayes wordmangle}
                      {interface: html}
                      {proxy: pop3}
                      {services:}

                      POPFile Engine v1.1.1 running

                      POPFile Engine v1.1.1 stopping

                      Stopping...

                      {classifier: bayes wordmangle}
                      {core: config history logger mq windows}
                      {interface: html}
                      {proxy: pop3}
                      {services:}

                      POPFile Engine v1.1.1 terminated


                      Status code: 0


                      (report finished 17-Mar-2010 @ 17:15:00)


                      which a] seems to be incomplete - no messages captured and b] doesn't strike me as particularly helpful.

                      Sorry about the length of this reply - and of this thread - I'm more and more tempted to clean the system down and start again.

                      All the best, and thanks for your efforts

                      Robert

                      • Message #1261

                        What are the values of the three "logger" entries in the ADVANCED page of the UI?


                        logger_format default
                        logger_level 2
                        logger_logdir /

                        I think that explains why you cannot see the log files.

                        logger_logdir should have the value ./ and then the log files should appear in the C:\Documents and Settings\Robert\Application Data\POPFile folder.

                        It might be worth posting your entire popfile.cfg file (shut down POPFile before you try to copy the file because POPFile always updates the file when it closes down) just in case something else is not quite right there.

                        I'll need to make a cup to tea before I read your reply properly.

                        Brian

                        • Message #1262

                          Brian

                          I think that explains why you cannot see the log files.

                          Sorry, but no it doesn't. When I said no log files are created I _meant_ no log files are created. No logs in the \%PROGRAM_FILES%\PR tree or in the \%SYSTEMROOT%\Docs&Settings\User\PF tree.

                          At this point I'm going to clean out the system and and registry and try again with a fresh install; there's clearly something fundamentally wrong here and I need to get this working sooner rather than later. I'll report back on the results.

                          Thanks for your patience.

                          Robert

                          • Message #1263

                            Robert,

                            I think you might have overlooked what Brian was trying to tell you. NOte the entry:

                            logger_logdir /

                            and Brian said that should read ./ not / that would mean you log files are in the root (/) not in ./ which would be the current working directory. you might weant to look further for the log files before getting so drastic.

                            bk

                            • Message #1264

                              BK

                              You're quite correct that I overlooked the period: none the less PF was not writing logfiles - in the drive root or anywhere else. I searched the system for files using 'log' as all or part of the filename: there were no PF logs anywhere. Which is extremely strange - and I have no explanation for that behaviour.

                              Since then I've restored the entire system to a state prior to installation of PF and AV: I then installed PF, modified the mail client connector to use the default PF settings - still with no AV and discovered that the results are identical: no X Headers are written to the mail received and no {string] prefix is written to the subject line or anywhere else in the message.
                              The diagnostics utilities show the same "clean" state, and PF _is_ classifying the inbound mail correctly.

                              Meanwhile the "old" machine continues to work perfectly with what is - on the surface - an identical system configuration.

                              I can only conclude that something about the state of a "clean" machine running XP/SP3 with all the latest hotfixes is interfering with PF operations. At this point I need to get on and deliver a working machine and can't afford to spend any more time trying to diagnose PF issues. Sadly, after many years of using PF successfully I'm going to have to find a different AS solution.

                              If I have spare time later I'll try to re-examine the problem.

                              Thanks to you and Brian and anyone else who's commented here.

                              Robert