POPFile forgot almost everything

[Ed Alford]

I've been using Popfile for several months now and up until recently it has been working very well. Now it seems to have forgotten all it has learned. It's clasifying emails wrongly or not at all. And it doesn't seem to want to learn anymore. I'm using it with Outlook 2010.

[Brian Smith]

POPFile uses an environment variable to find its data files. If it cannot find its data files then it will revert to some default values and this means it can appear to forget everything it had learned.

The Windows version of POPFile comes with a diagnostic utility that checks some important POPFile settings, including the environment variable that is supposed to point to the data files.

The Windows installer creates a Start Menu shortcut which runs the diagnostic utility:

Start -- All Programs -- POPFile -- Support -- PFI Diagnostic utility (full)

(Do not use the "PFI Diagnostic utility (simple)" shortcut)

The diagnostic utility displays a report in a scrollable window; if you right-click inside the window you can copy the report to the clipboard and paste it into a reply.

[Ed Alford]

---

POPFile PFI Diagnostic Utility v0.3.0 (full mode)

---

String data report format (not used for numeric data)

string not found : >< empty string found : < > string with 'xyz' value found : < xyz >

---

Current UserName = Ed (User)

Operating System = Windows 7 Professional (64-bit) IsNT return code = 1 Internet Explorer = 9.1

---

Location used to store temporary files

---

$TEMP folder path = < C:\Users\Ed\AppData\Local\Temp >

---

Start Menu Locations

---

AU: $SMPROGRAMS = < C:\ProgramData\Microsoft\Windows\Start Menu\Programs > AU: $SMSTARTUP = < C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup >

Search results for the "AU: $SMSTARTUP" folder:

*.lnk files found = 0 POPFile shortcuts = 0

CU: $SMPROGRAMS = < C:\Users\Ed\AppData\Roaming\Microsoft\Windows\Start Menu\Programs > CU: $SMSTARTUP = < C:\Users\Ed\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup >

Search results for the "CU: $SMSTARTUP" folder:

*.lnk files found = 1 POPFile shortcuts = 0

---

Obsolete/testbed Registry Entries

---

[1] Pre-0.21 Data:

Pre-0.21 POPFile = >< Pre-0.21 Testbed = ><

[2] 0.21 Test Installer Data:

HKLM: RootDir_LFN = >< HKLM: RootDir_SFN = ><

HKCU: RootDir_LFN = >< HKCU: RootDir_SFN = >< HKCU: UserDir_LFN = >< HKCU: UserDir_SFN = ><

[3] Current PFI Testbed Data:

MRI PFI Testbed = >< MRI PFI Testdata = ><

---

POPFile Registry Data

---

NTFS SFN Disabled = < 2 >

HKLM: MRI Version = < 1.1.3 >

HKLM: NewParser = ><

HKLM: InstallPath = < D:\POPFile > HKLM: RootDir_LFN = < D:\POPFile > HKLM: RootDir_SFN = < D:\POPFile >

HKLM: *.exe count = 6 (this is OK)

HKCU: Data Owner = < Ed > HKCU: MRI Version = < 1.1.3 > HKCU: RootDir_LFN = < D:\POPFile > HKCU: RootDir_SFN = < D:\POPFile >

HKCU: UserDir_LFN = < C:\Users\Ed\AppData\Roaming\POPFile > HKCU: UserDir_SFN = < C:\Users\Ed\AppData\Roaming\POPFile >

HKCU: popfile.pl = found HKCU: popfile.cfg = found

HKCU: *.exe count = 6 (this is OK)

---

POPFile Logger Settings

---

Logger output = < 1 > (To File) Logger format = < default > Logger level = < 0 > Logger directory = < ./ > (C:\Users\Ed\AppData\Roaming\POPFile)

---

POPFile Environment Variables

---

'POPFILE_ROOT' = < D:\POPFile > 'POPFILE_USER' = < C:\Users\Ed\AppData\Roaming\POPFile >

Env: popfile.pl = found Env: popfile.cfg = found

ROOT: *.exe count = 6 (this is OK)

'ITAIJIDICTPATH' = >< (this is OK) 'KANWADICTPATH' = >< (this is OK)

'MECABRC' = >< (this is OK)

---

POPFile Configuration Settings (subset)

---

POPFile UI port = < 8080 > POP3 Listen port = < 110 > Concurrent POP3 = < 0 >

---

(report created 26-Sep-2013 @ 19:19:44)

---

[Brian Smith]

Thank you for the diagnostic report.

The diagnostic utility did not find a StartUp shortcut for POPFile. How are you starting POPFile?

The diagnostic utility did not find any problems with the environment variables so it looks like something else has gone wrong.

If POPFile's database gets corrupted then strange things can happen. The Windows version of POPFile comes with a utility which checks the integrity of the database. Shut down POPFile then use this Start Menu shortcut to check the database:

Start -- All Programs -- POPFile -- Support -- Check database status

This utility will not be able to check the database if POPFile is already running. A report will be generated in a scrollable window and can be pasted into a reply, just like the diagnostic utility's report.

If the database is OK then it is possible that the problem is with the connection between Outlook 2010 and POPFile.

It's classifying emails wrongly or not at all. And it doesn't seem to want to learn anymore.

The HISTORY page in POPFile's User Interface should list all incoming POP3 messages handled by POPFile.

Are you saying that some messages are not appearing here?

Are you saying that attempts to reclassify messages via the HISTORY page now fail?

The BUCKETS page has a statistics section showing the number of messages processed etc ~ are these counts no longer being updated correctly?

The diagnostic utility's report shows you are using the default POP3 port (110). Have you recently changed or updated your anti-virus package? If something has gone wrong with the POP3 proxy chain this will affect the connection between Outlook 2010 and POPFile.

The diagnostic utility's report shows that you are using the default logger level. Try increasing the logger level to 1 as this will show how many email messages POPFile is classifying. The "Preparing a log to help debugging problems" page in the wiki explains how to change the logger_level and the "Logger Level" page has some log extracts.

Although logger level 2 provides more information in the log file, it can be harder to interpret the log file as a lot of information gets recorded whenever the User Interface is accessed (and this makes it harder to see how POPFile's POP3 processing is behaving).

By default POPFile does not display any Perl warnings or error messages. To see these hidden messages you need to shut down POPFile and then use this Start Menu shortcut to start POPFile:

Start -- All Programs -- POPFile -- Support -- Message Capture utility

This will use your normal settings to start POPFile and display a scrollable window showing the Perl messages which are normally hidden. Once you see the message "POPFile Engine v1.1.3 running" in that Window POPFile is ready to handle email.

Do any warning or error messages appear in the Message Capture utility's window when POPFile starts up or when Outlook 2010 tries to check for new mail? You can right-click in the window to copy the contents to the clipboard, just like the diagnostic and database checking utilities.

[Ed Alford]

Popfile starts automatically at system start up.

Here is the results of the database test:

---

POPFile SQLite Database Status Check (integrated) v0.2.1

---

Current user : Ed Current folder: D:\POPFile Command line : /REGISTRY

Trying to find database using registry data (HKCU)... found it!

POPFile database found (C:\Users\Ed\AppData\Roaming\POPFile\popfile.db)

SQLite v3.7.2 utility found in D:\POPFile

Database is in SQLite 3.x format, uses schema version 3 and its size is 2,206 KB

Result of running the 'pragma integrity_check;' command: * in database main * Page 968 is never used Page 1207 is never used Page 1219 is never used Page 1220 is never used Page 1221 is never used Page 1222 is never used rowid 15681 missing from index sqlite_autoindex_words_1 rowid 15682 missing from index sqlite_autoindex_words_1 rowid 15683 missing from index sqlite_autoindex_words_1 rowid 15684 missing from index sqlite_autoindex_words_1 rowid 15685 missing from index sqlite_autoindex_words_1 rowid 15686 missing from index sqlite_autoindex_words_1 rowid 15687 missing from index sqlite_autoindex_words_1 rowid 15688 missing from index sqlite_autoindex_words_1 rowid 15689 missing from index sqlite_autoindex_words_1 rowid 15690 missing from index sqlite_autoindex_words_1 rowid 15691 missing from index sqlite_autoindex_words_1 rowid 15692 missing from index sqlite_autoindex_words_1 rowid 15693 missing from index sqlite_autoindex_words_1 rowid 15694 missing from index sqlite_autoindex_words_1 rowid 15695 missing from index sqlite_autoindex_words_1 rowid 15696 missing from index sqlite_autoindex_words_1 wrong # of entries in index sqlite_autoindex_words_1 rowid 23861 missing from index sqlite_autoindex_matrix_1 rowid 23886 missing from index sqlite_autoindex_matrix_1 rowid 23892 missing from index sqlite_autoindex_matrix_1 rowid 23897 missing from index sqlite_autoindex_matrix_1 rowid 23900 missing from index sqlite_autoindex_matrix_1 rowid 23913 missing from index sqlite_autoindex_matrix_1 wrong # of entries in index sqlite_autoindex_matrix_1

The POPFile database has passed the SQLite integrity check!

---

(report finished 27-Sep-2013 @ 19:07:37)

---

Some of the emails are classified wrong but most of them are now unclassified.

All messages are appearing in the history. Just most are unclassified.

I can reclassify messages via the history page but it doesn't seem to stick for the next round of emails coming in.

The bucket statistics were reset last Friday and now they don't seem to be updating. It only shows 4 unclassified messages since then and almost all of them now are coming in as unclassified.

I started the message capture utility and sent myself an email. It classified it correctly as personal but there was an error in the log:

---

POPFile Message Capture Utility v0.2.0

---

POPFILE_ROOT = D:\POPFile POPFILE_USER = C:\Users\Ed\AppData\Roaming\POPFile Using 'popfileif.exe' to run POPFile

---

(report started 27-Sep-2013 @ 19:11:59)

---

POPFile Engine loading

Loading...

{core: windows} {core: config history logger mq} {classifier: bayes wordmangle} {interface: html} {proxy: pop3} {services:}

POPFile Engine v1.1.3 starting

Initializing...

{core: config history logger mq windows} {classifier: bayes wordmangle} {interface: html} {proxy: pop3} {services:}

Starting...

{core: config history logger mq windows} {classifier: bayes wordmangle} {interface: html} {proxy: pop3} {services:}

POPFile Engine v1.1.3 running DBD::SQLite::st fetchrow_arrayref failed: database disk image is malformed at D:\POPFile/Classifier/Bayes.pm line 2473, <GEN19> line 7.

Does this mean the database is corrupt? If so, what can I do to correct it?

[Brian Smith]

Popfile starts automatically at system start up.

Although the diagnostic utility found one StartUp shortcut (for the 'Ed' account) it did not think this shortcut started POPFile.

What is the name of this shortcut and what is its "Target" ? The "Target" is one of the fields shown if you right-click the shortcut and select "Properties". The diagnostic utility uses a simple test to detect POPFile shortcuts ~ perhaps a better test is needed here?

Here is the results of the database test

By default POPFile uses a SQLite 3.x format database. The "Check database status" shortcut uses the standard SQLite 3.x command-line utility to check if the database is a valid SQLite 3.x format database (it does not run any POPFile-specific checks on the content of the database).

The "The POPFile database has passed the SQLite integrity check!" line means that the SQLite command-line utility did not return an error code. However some unexpected messages were produced, indicating that some problems were found. I hope these messages are only warnings.

DBD::SQLite::st fetchrow_arrayref failed: database disk image is malformed at D:\POPFile/Classifier/Bayes.pm line 2473, <GEN19> line 7.

This entry in the Message Capture utility's report means there is a more serious problem. If POPFile cannot access the database properly this would explain the symptoms you are seeing.

Does this mean the database is corrupt? If so, what can I do to correct it?

The SQLite database is a single file (C:\Users\Ed\AppData\Roaming\POPFile\popfile.db).

Before trying to fix the problem, shut down POPFile and make a copy of this file (called damaged-popfile.db or something similar) just in case the fix makes things worse and you want to undo the fix.

If you have a backup copy of the database file you can simply shut down POPFile (to make sure it is not using the database) and replace the popfile.db file with an older one. For example, if you use Windows Backup then right-clicking popfile.db and selecting "Restore previous versions" might let you restore an undamaged copy of the database.

If you don't have a suitable backup copy, the SQLite 3.x command-line utility may be able to rebuild (most of?) the database for you.

The official SQLite web site says "Depending how badly your database is corrupted, you may be able to recover some of the data by using the CLI to dump the schema and contents to a file and then recreate."

On my system it only took a few seconds to rebuild my error-free 3,373 KB database using the SQLite command-line utility.

You'll need to open a command window at your POPFile User Data folder and type in some DOS commands. (I don't know if you have any experience with this sort of thing, so if you prefer you could send me (brian at getpopfile.org) your damaged popfile.db file and I can try rebuilding it for you.)

The diagnostic report means I can tell you the exactly what to type.

(1) If the command window is opened at the correct folder you should see the prompt:

C:\Users\Ed\AppData\Roaming\POPFile>

(2) Use the SQLite 3.x command-line utility to export the entire contents of the damaged database into a single ASCII file by typing the following command

D:\POPFile\sqlite3.exe popfile.db .dump > damaged.dump

(3) After pressing the ENTER key to execute that command there may be a delay before the command prompt returns:

C:\Users\Ed\AppData\Roaming\POPFile>

(4) Use the SQLite 3.x command-line utility to build a new database file using the new ASCII file by entering the following command

D:\POPFile\sqlite3.exe new-popfile.db < damaged.dump

(5) After pressing the ENTER key to execute that command there may be a delay before the command prompt returns:

C:\Users\Ed\AppData\Roaming\POPFile>

(6) Now check the integrity of the newly created database by entering the following command

D:\POPFile\pfidbstatus.exe new-popfile.db

(7) After pressing the ENTER key to execute that command the "POPFile SQLite Database Status Check" utility's window will appear and, I hope, it will show

Result of running the 'pragma integrity_check;' command:
ok

The POPFile database has passed the SQLite integrity check!

The new database file will probably be a bit smaller than the damaged one as the rebuild process also defragments the database.

To make POPFile use the new database, rename it as popfile.db. To double-check that POPFile will use the new database, use the Start Menu's "Check database status" shortcut and it should report the new database size.

[Ed Alford]

Well, I restored my Popfile.db back to the first part of this month. Then I ran an integrity check here it is:

---

POPFile SQLite Database Status Check (integrated) v0.2.1

---

Current user : Ed Current folder: D:\POPFile Command line : /REGISTRY

Trying to find database using registry data (HKCU)... found it!

POPFile database found (C:\Users\Ed\AppData\Roaming\POPFile\popfile.db)

SQLite v3.7.2 utility found in D:\POPFile

Database is in SQLite 3.x format, uses schema version 3 and its size is 1,213 KB

Result of running the 'pragma integrity_check;' command: ok

The POPFile database has passed the SQLite integrity check!

---

(report finished 28-Sep-2013 @ 19:02:12)

---

I will see if things straighten out before I try the doss commands.

[Ed Alford]

I forgot to add the start up info at system start up: D:\popfile\runPOPFile.exe