Open Discussion → POPFile command line hack
Hi, I used popfile a while back and I like the web interface. Would it be possible to write a command line frontend like bogofilter, to input emails as files and detect spam base on an exit code of the script?
-
Message #1315
POPFile's XML-RPC interface may be what you are looking for:
brian06/01/10 15:58:00 -
-
Message #1316
thanks, the xmlrpc look really cool but i am not sure ill be able to install the xmlrpc on my server. can this be done using LWP::UserAgent??
or using one of the pm's like this?
use lib qw(/home/popfile/Classifier/);
use MailParse?;
use Bayes;
joeharth06/02/10 02:31:03 -
-
Message #1317
What are you trying to do? Without knowing what you are trying to achieve with POPFile, it is almost impossible to provide any useful answers.
If all you want to do is use the command-line to classify a message then the pipe.pl script (which is included with POPFile) can be used:
brian06/02/10 15:13:44 -
-
Message #1318
Thanks Brian, I was able to write 3 scripts. One for ham one, for spam, and one for results. I commented out the ARG part of the script and added something like this for ham
$outmsg = $b->add_message_to_bucket( $session, 'ham', $ARGV[0] )
and something like this for results
$outmsg = $b->classify( $session, $ARGV[0] );
looks like the script will somehow conflict when the interface runs or for some reason it wont display words learn until I restart the web interface.
I also did not see any of the learned messages, just keywords. I have my mailserver in learning mode. It should learn words in the next few hours.
Currently I use bmf, bogofilter and qsf as part of my script. Hopefully my popfile hack will be the only thing that I will need.
joeharth06/02/10 22:25:30 -
-
Message #1319
I also did not see any of the learned messages
pipe.pl is only one of the Utility Scripts Distributed with POPFile
POPFile's Message History is used to correct any mistakes made by POPFile when it classifies your email.
The pipe.pl script lets POPFile decide how to classify a particular message. This message will appear in the Message History to let you change the classification.
The insert.pl script tells POPFile how to classify a particular message. Therefore there is no need for this message to appear in the Message History (because POPFile did NOT decide how to classify it).
brian06/03/10 14:42:08 -
-
Message #1320
Wow I guess I did not have to write anything. I can use insert.pl instead. I was unable to use the script outside the popfile folder so I added
use lib '/home/popfile';
chdir '/home/popfile' or die "Can't cd to popfile: $!\n";
I still see nothing in the history table unless I pop emails. I wrote a little script to output the history table. I can see email that are popped but no emails that I run through pipe.pl. I do cat email.txt|./pipe.pl and the output seems to work
!/usr/bin/perl -w
use DBI;
use strict;
my $db = DBI->connect("dbi:SQLite:popfile.db", "", "",
{RaiseError? => 1, AutoCommit? => 1});
#$db->do("CREATE TABLE n (id INTEGER PRIMARY KEY, f TEXT, l TEXT)");
#$db->do("INSERT INTO n VALUES (NULL, 'john', 'smith')");
my $all = $db->selectall_arrayref("SELECT * FROM history");
my $lastn;
foreach my $row (@$all) {
my ($a, $b, $c, $d, $e, $f, $g, $h, $i) = @$row;
print "$a|$b|$c|$d|$e|$f|$g|$h|$i\n";
}
joeharth06/03/10 17:33:32 -
-
Message #1321
Hi Brian, I found a problem but I also found the work around.
pipe should be
$b->classify_and_modify( $session, \*STDIN, \*STDOUT, 0, , 0, 1, "\n" );
instead of
$b->classify_and_modify( $session, \*STDIN, \*STDOUT, 1, , 0, 1, "\n" );
after enabling history I had to force bucketid to be 1 (unclassfied) in History.pm
for some reason I get "Couldn't find bucket ID for bucket"
not very clean but it worked
# if ( defined( $bucketid ) ) {
my $result = $update_history->execute( # PROFILE BLOCK START
${$header{from}}[0], # hdr_from
${$header{to}}[0], # hdr_to
${$header{date}}[0], # hdr_date
${$header{cc}}[0], # hdr_cc
${$header{subject}}[0], # hdr_subject
$sort_headers{from}, # sort_from
$sort_headers{to}, # sort_to
$sort_headers{cc}, # sort_cc
1, # committed
1, # bucketid
0, # usedtobe
$magnet, # magnetid
$hash, # hash
$msg_size, # size
$slot # id
); # PROFILE BLOCK STOP
# } else {
# $self->log_( 0, "Couldn't find bucket ID for bucket $bucket when committing $slot" );
# $self->release_slot( $slot );
# }
joeharth06/03/10 22:08:06 -
-
Message #1322
I still see nothing in the history table unless I pop emails
Sorry, I was wrong about that. I'd been using the same test message with my XML-RPC script and had looked at the wrong UI when I was writing my reply. I have over 20 separate POPFile installations on my computer so it is easy for me to get confused! Looks like it is time to delete most of them; I do this every few months (usually after I get confused).
When I re-installed POPFile and tested pipe.pl again I found that the messages did not appear in the HISTORY page.
I also found the work around ... for some reason I get "Couldn't find bucket ID for bucket"
I found the same thing when I tried to disable the "nosave" option to make the message appear in the HISTORY. I think the problem is to do with the way the session key is handled (it gets released too soon?) but I don't know Perl so I am not sure what to do about it:
4820: mq: 380: post COMIT (ObU0D4aye65AYDwEke:1:advanced_cdr:0) 4820: mq: 387: queuing post COMIT (ObU0D4aye65AYDwEke:1:advanced_cdr:0) 4820: mq: 389: COMIT queue length now 0 4820: mq: 380: post RELSE (ObU0D4aye65AYDwEke) 4820: mq: 387: queuing post RELSE (ObU0D4aye65AYDwEke) 4820: mq: 389: RELSE queue length now 1 4820: mq: 128: Message COMIT (ObU0D4aye65AYDwEke:1:advanced_cdr:0) ready for delivery 4820: mq: 131: Delivering message COMIT (ObU0D4aye65AYDwEke:1:advanced_cdr:0) to history 4820: mq: 131: Delivering message COMIT (ObU0D4aye65AYDwEke:1:advanced_cdr:0) to bayes 4820: mq: 128: Message RELSE (N0sCNMMOwBY872sIo) ready for delivery 4820: mq: 131: Delivering message RELSE (N0sCNMMOwBY872sIo) to bayes 4820: bayes: 2087: release_session_key releasing key N0sCNMMOwBY872sIo for user XXXXXX 4820: mq: 128: Message RELSE (ObU0D4aye65AYDwEke) ready for delivery 4820: mq: 131: Delivering message RELSE (ObU0D4aye65AYDwEke) to bayes 4820: bayes: 2087: release_session_key releasing key ObU0D4aye65AYDwEke for user XXXXXX 4820: bayes: 2123: Invalid session key ObU0D4aye65AYDwEke provided in Classifier::Bayes @ 3245 4820: history: 675: Couldn't find bucket ID for bucket advanced_cdr when committing 1 4820: POPFile stopped
brian06/04/10 01:39:22 -
-
Message #1323
I think the problem is to do with the way the session key is handled ... but I don't know Perl so I am not sure what to do about it
After some experimentation with a modified version of pipe.pl called pipe-history.pl I think I've found a solution:
5188: mq: 380: post COMIT (qxaYaWoiC2sH8uAA35:10:spam:0) 5188: mq: 387: queuing post COMIT (qxaYaWoiC2sH8uAA35:10:spam:0) 5188: mq: 389: COMIT queue length now 0 5188: mq: 128: Message COMIT (qxaYaWoiC2sH8uAA35:10:spam:0) ready for delivery 5188: mq: 131: Delivering message COMIT (qxaYaWoiC2sH8uAA35:10:spam:0) to history 5188: mq: 131: Delivering message COMIT (qxaYaWoiC2sH8uAA35:10:spam:0) to bayes 5188: mq: 128: Message RELSE (Aq6cYIlASsVS38r7HSA2) ready for delivery 5188: mq: 131: Delivering message RELSE (Aq6cYIlASsVS38r7HSA2) to bayes 5188: bayes: 2087: release_session_key releasing key Aq6cYIlASsVS38r7HSA2 for user XXXXXX 5188: POPFile stopped
When I was investigating this I found that POPFile's configuration data was being corrupted when I ran the utility script.
It turns out that Naoki has found a problem with some of the utility scripts (see Configuration changes do not survive reboot) and has made the the necessary changes (see changeset:3661).
The current POPFile release does NOT include these fixes.
Brian
brian06/10/10 20:20:49 -
-
Message #1537
Hi, I'm so sorry about this serious thread-necromancy but I feel that it's better to post in this thread than to create a new one.
After some experimentation with a modified version of pipe.pl called pipe-history.pl I think I've found a solution.
Could you share this pipe-history.pl? pipe.pl is the most convenient way for me to use POPFile but the lack of the History (and thus the way to correct the mail's assignment to a bucket) is very frustrating.
Or should I just use insert.pl for messages that were evaluated incorrectly? I don't know if it does the same like reclassifying the e-mail via web UI...
bezda08/14/11 23:52:07 -
-
Message #1538
Could you share this pipe-history.pl?
The file is still on my hard disk but it is over a year since I last used it. I want to check it still works with POPFile 1.1.2 which will be released soon.
I only changed a few lines in pipe.pl to make pipe-history.pl so it should not take long to test the modified script with the new POPFile 1.1.2 code.
Brian
brian08/15/11 13:04:41 -
-
Message #1546
My modified 'pipe.pl' script works in POPFile 1.1.1 and 1.1.2 so I have, at last, made it available.
The 'pipe-history.pl' script has been attached to ticket:164 (Modified version of 'pipe.pl' which updates the message history) because the forum software does not support attachments.
Here is a list of the changes I made to the 'pipe.pl' script after Naoki updated it to fix the 'popfile.cfg' corruption problem:
4,5c4,5 < # pipe.pl --- Read a message in on STDIN and write out the modified < # version on STDOUT --- > # pipe-history.pl --- Read a message in on STDIN and write out the modified > # version on STDOUT. Save message in message history to permit reclassification. 71c71,73 < $b->classify_and_modify( $session, \*STDIN, \*STDOUT, 1, '', 0, 1, "\n" ); --- > # Save this message in the message history, unlike the 'pipe.pl' script > > $b->classify_and_modify( $session, \*STDIN, \*STDOUT, 0, '', 0, 1, "\n" ); 79c81 < $b->release_session_key( $session ); --- > # $b->release_session_key( $session ); 85,86c87,88 < print "pipe.pl - reads a message on STDIN, classifies it, outputs the modified version on STDOUT\n\n"; < print "Usage: pipe.pl\n"; --- > print "Classify a msg read on STDIN, update the history & send modified msg to STDOUT\n\n"; > print "Usage: pipe-history.pl\n";
This DIFF file is also attached to ticket:164 (Modified version of 'pipe.pl' which updates the message history)
Brian
brian08/20/11 18:24:31
-
-
-
-
-
-
-
-
-
-
-
