How to build the Windows installer for POPFile 1.1.2 (draft)

The Community Edition (i.e. the free edition) of ActivePerl 5.8.x is no longer available

Introduction

These instructions describe how to use a Windows system to build the POPFile Windows installer and the NSIS-based POPFile utilities for a new POPFile release. These notes record the steps taken to create the POPFile 1.1.2 release from scratch (i.e. on a system not previously used to build the installer).

Due to the way in which the NSIS compiler works, building an installer using the files used for an official POPFile release will NOT result in an identical installer program (setup.exe).

Overview

  • Install third-party software:
    • ActivePerl
    • Perl Dev Kit (PDK) (optional)
    • NSIS Compiler
    • Additional NSIS plugins
  • Obtain the source files from POPFile's Subversion repository
  • Obtain the Kakasi package (used to analyze Japanese text)
  • Obtain the relevant SQLite command-line utilities
  • Install Make and 7-Zip (anything else?)
  • Create the missing files (two change note files, anything else?)
  • Build the Perl-based programs (optional)
  • Build the POPFile Windows installer

Install third-party software

ActivePerl

Installation

The Windows version of POPFile uses a minimal Perl based upon ActiveState's ActivePerl. Although Perl 5.10.x, 5.12.x and 5.14.x have been released, POPFile 1.x still uses Perl 5.8.x. When work started on the POPFile 1.1.2 release the most recent ActivePerl 5.8.x Community Edition release was ActivePerl 5.8.9 Build 829.

The Windows version of POPFile is a 32-bit program so the Windows Installer (MSI) file for the Windows (x86) version of ActivePerl 5.8.x is required. When I used the MSI installation package I disabled the Perl ISAPI, PerlEx and PerlScript options. I performed a 'clean' install, I did not upgrade an existing ActivePerl installation.

ActiveState only provide Community Edition (i.e. free) download links for the current builds of the Windows, Linux and Mac OS X versions of ActivePerl. 1) This means there may be some differences between the minimal Perl used in an official release and the minimal Perl produced using the current Community Edition of ActivePerl. If you wish to build an installer that installs an identical minimal Perl to that used for a particular POPFile release the safest way to do this is by “reverse engineering” the minimal Perl from that release.

FIXME Explain how to “reverse engineer” the minimal Perl from a POPFile release.

The source files used to build the POPFile installer assume that ActivePerl is installed in the default location (C:\Perl) but they can be easily reconfigured if ActivePerl is installed in a different location. 2)

PPM Repositories

By default the Perl Package Manager (PPM) does not display the Repo column. It is recommended that the Repo column is selected in the View Columns list (under the View menu in PPM) as this makes it easy to see the source repository for each module listed.

By default PPM is configured to use the ActiveState repository. POPFile requires some additional Perl modules which are not included in this repository therefore PPM has to be configured to use some additional repositories. Add the following repositories from the drop-down list provided by PPM:

Repo Code Repository Name Packages Required for POPFile
bribes bribes :: Bribes de Perl Net-SSLeay package
tcool tcool :: Kenichi Ishigaki's repository IO-Socket-SSL package
uwinnipeg uwinnipeg :: University of Winnipeg Text-Kakasi package

If a particular Perl module is available from more than one repository the POPFile Project has always preferred to use the ActiveState repository as the source.

Perl Updates

Use the Perl Package Manager (PPM) to apply all of the available updates to the ActivePerl installation. PPM may install additional packages not mentioned in the list of updates in order to meet the dependency requirements of the packages which are being updated.

When PPM updates an existing Perl package or installs a new Perl package these changes are stored separately from the packages included in the ActivePerl distribution. This means that the Windows installer uses a mixture of original and new/updated package paths when assembling the minimal Perl.

Here is a complete list of the PPM updates applied to a clean installation of ActivePerl 5.8.9 build 829 for the POPFile 1.1.2 release. Each package was updated separately (i.e. the updates were done one package at a time).

Package Name Version Repository Description
Archive-Extract 0.52 ActiveState Generic archive extracting mechanism
Attribute-Handlers 0.91 ActiveState Simpler definition of attribute handlers
common-sense 3.4 ActiveState save a tree AND a kitten, use common::sense!
Compress-Raw-Bzip2 2.035 ActiveState Low-Level Interface to bzip2 compression library
Compress-Raw-Zlib 2.035 ActiveState Low-Level Interface to zlib compression library
constant 1.21 ActiveState Perl pragma to declare constants
DBD-ODBC 1.27 ActiveState ODBC DBD for Perl DBI
Digest-SHA 5.62 ActiveState Perl extension for SHA-1/224/256/384/512
Encode 2.43 ActiveState character encodings in Perl
ExtUtils-CBuilder 0.280202 ActiveState Compile and link C code for Perl modules
ExtUtils-Command 1.17 ActiveState utilities to replace common UNIX commands in Makefiles etc.
File-Fetch 0.32 ActiveState Generic file fetching code
File-HomeDir 0.97 ActiveState Find your home and other directories on any platform
Filter 1.39 ActiveState Source Filters
Filter-Simple 0.87 ActiveState Simplified source filtering
HTML-Tree 4.2 ActiveState Class for objects that represent HTML elements
IO-Compress 2.035 ActiveState IO Interface to compressed data files/buffers
IPC-Cmd 0.70 ActiveState A cross platform way of running (interactive) commandline programs.
JSON 2.53 ActiveState JSON (JavaScript Object Notation) encoder/decoder
JSON-XS 2.3 ActiveState null
Locale-Codes 3.16 ActiveState a distribution of modules to handle locale codes
Log-Message-Simple 0.08 ActiveState Simplified interface to Log::Message
Math-BigInt 1.993 ActiveState Arbitrary size integer/float math package
Math-BigInt-FastCalc 0.28 ActiveState Math::BigInt::Calc with some XS for more speed
Math-BigRat 0.2602 ActiveState Arbitrary big rational numbers
Math-Complex 1.57 ActiveState complex numbers and associated mathematical functions
Module-Build 0.3624 ActiveState Build and install Perl modules
Module-Metadata 1.000004 ActiveState Gather package and POD information from perl module files (automatically installed by PPM when the 'Module-Build 0.3624' package was installed)
version 0.88 ActiveState Structured version objects (automatically installed by PPM when the 'Module-Build 0.3624' package was installed)
Perl-OSType 1.002 ActiveState Map Perl operating system names to generic types (automatically installed by PPM when the 'Module-Build 0.3624' package was installed )
CPAN-Meta-YAML 0.003 ActiveState Read and write a subset of YAML for CPAN Meta files (automatically installed by PPM when the 'Module-Build 0.3624' package was installed )
Module-Corelist 2.49 ActiveState what modules shipped with versions of perl
Module-Load-Conditional 0.44 ActiveState Looking up module information / loading at runtime
Object-Assessor 0.42 ActiveState Per object accessors
Params-Util 1.04 ActiveState Simple, compact and correct param-checking functions
parent 0.225 ActiveState Establish an ISA relationship with base classes at compile time
Pod-Simple 3.16 ActiveState framework for parsing Pod
SOAP-Lite 0.712 ActiveState Perl's Web Services Toolkit (generates an error message about Apache::)
SQL-Statement 1.33 ActiveState SQL parsing and processing engine
Storable 2.25 ActiveState persistence for Perl data structures
Term-UI 0.26 ActiveState User interfaces via Term::ReadLine made easy
Test-Differences 0.61 ActiveState Test strings and data structures and show differences if not ok
Test-Harness 3.23 ActiveState Run Perl standard test scripts with statistics
Test-Simple 0.98 ActiveState Basic utilities for writing tests.
Text-CSV_XS 0.82 ActiveState Comma-Separated Values manipulation routines
Text-Diff 1.41 ActiveState Perform diffs on files and record sets
Text-Glob 0.09 ActiveState match globbing patterns against text
threads 1.83 ActiveState Perl interpreter-based threads
threads-shared 1.37 ActiveState Perl extension for sharing data structures between threads
Time-HiRes 1.9722 ActiveState High resolution alarm, sleep, gettimeofday, interval timers
Unicode-Normalize 1.07 ActiveState Unicode Normalization Forms
URI 1.58 ActiveState Uniform Resource Identifiers (absolute and relative)
Win32-API 0.62 ActiveState Perl Win32 API Import Facility
Win32API-Registry 0.32 ActiveState Low-level access to Win32 system API calls from WINREG.H
YAML-LibYAML 0.35 ActiveState

Additional Perl packages

POPFile requires some additional Perl packages which are not part of the default ActivePerl installation. Use PPM to install the following extra Packages:

Package Name Version Repository Description
Class-Inspector 1.25 ActiveState Get information about a class and its structure
File-Glob-Windows 0.1.4 ActiveState glob routine for Windows environment
IO-Socket-Socks 0.4 ActiveState Provides a way to create socks client or server both 4 and 5 version
IO-Socket-SSL 1.13 tcool Nearly transparent SSL encapsulation for IO::Socket::INET
Net-SSLeay 1.36 bribes Perl extension for using OpenSSL (automatically installed by PPM when the 'IO-Socket-SSL 1.13' package was installed)
Text-Kakasi 2.04 uwinnipeg perl frontend to kakasi
TimeDate 1.20 ActiveState Parsing of date strings
Win32-GUI 1.06 ActiveState Perl-Win32 Graphical User Interface Extension

POPFile has used a SQLite 3.x format database by default since the 1.1.0 release in November 2008. To allow POPFile to upgrade databases used by older versions of POPFile, two more packages are required:

Package Name Version Description
BerkeleyDB 0.22 Perl extension for BerkeleyDB
DBD-SQLite2 0.33 Self-contained RDBMS in a DBI driver (sqlite2.x)

The BerkeleyDB package is used when upgrading a POPFile 0.20.x installation. An old version of this package is required in order to be able to upgrade databases created by POPFile 0.20.x (otherwise the database upgrade loses all of the old data from the BerkeleyDB database files!).

The DBD-SQLite2 package is used when upgrading an old SQLite 2.x format database (as used by POPFile 0.21.0 to 1.0.1).

Since both of these packages are old versions it seems that an ActiveState Business Edition license has to be purchased in order to download the binary versions from ActiveState. However the package source files are still available from CPAN ( BerkeleyDB-0.22.tar.gz and DBD-SQLite2-0.33.tar.gz).

FIXME Give download links for the BerkeleyDB-0.22-modified.zip and DBD-SQLite2-0.33.zip files plus installation instructions for these files.

FIXME Give instructions for using PPM's command-line mode to install BerkeleyDB v-.22 and DBD-SQLite2 v0.33

PDK - Perl Dev Kit (optional)

The Windows version of POPFile ships with six Perl-based programs:

  • popfile.exe (the main POPFile executable, starts one of the following four programs)
    • popfileb.exe (runs POPFile in the background)
    • popfilef.exe (runs POPFile in a “DOS-box” console window)
    • popfileib.exe (as popfileb.exe plus a system tray icon)
    • popfileif.exe (as popfilef.exe plus a system tray icon)
  • popfile-service.exe (runs POPFile as a service)

These programs are all built using ActiveState's Perl Dev Kit (PDK). The Windows version of POPFile is a 32-bit program so the 32-bit (x86) Windows version of the PDK is required. Unlike ActivePerl, the PDK requires a licence. In June 2011 the cost of a single-user PDK licence was US $295. A time-limited free trial version of the PDK is also available.

Note that POPFile's Subversion repository includes pre-built versions of these six Perl-based programs which is why the PDK is described as “optional” here. The Subversion log entry for each program normally identifies the PDK version used to build the program.

PDK 9.0.1 build 293382 (released on 9 August 2010) was used to build these six programs for the POPFile 1.1.2 release.

NSIS Compiler

The Windows installer for POPFile and several of the Windows utilities shipped with POPFile are built using NSIS, the Nullsoft Scriptable Install System. NSIS was SourceForge “Project of the Month” for January 2006 (POPFile was “Project of the Month” for May 2003).

The source files for the installer and the NSIS-based utilities all mention the version of the NSIS compiler which should be used (e.g. lines 38 to 59 of the app-vcheck.nsi file). If a different version of the compiler is used then a suitable warning message will appear in the compiler log.

At the time of writing POPFile uses version 2.45 of the NSIS compiler which can be downloaded from SourceForge using this link:

http://prdownloads.sourceforge.net/nsis/nsis-2.45-setup.exe?download

After NSIS is installed the compiler can be run using the GUI front-end, the command-line version or via the context menu for NSIS script files.

The POPFile Makefile uses the command-line version of the compiler (makensis.exe) and it is easier to run this version if you update the PATH to include the folder containing this file.

Additional NSIS Plugins

The POPFile installer and some of the NSIS-based POPFile utilities require some additional NSIS plugins. Each plugin is usually made available as a zip file containing the plugin DLL, the source code for the DLL, some documentation, one or more example scripts and occasionally some macros and other support functions in a NSIS header (*.nsh) file.

Plugin Link Version used by POPFile 1.1.2 Description
AccessControl 5 November 2009 (current version) Access Control List (ACL) management
DumpLog v1.0 (current version) Dump the log of the installer (installer details) to a file
dumpstate v0.2 (current version) Displays the current state of the NSIS variables and stack 3)
getsize v1.0 (current version) Finds size of a file or folder, counts files in folder
GetVersion v1.5 (current version) Windows version information (name, type, version, etc)
inetc 28 April 2011 (current version) Internet client plug-in for files download and upload
LockedList v2.3 (current version) Gets a list of programs that are locking a selection of files
md5dll v0.5 (current version) Generates a md5 value from contents of specified file
MoreInfo v1.0.1.2 (current version) Retrieves version information from files
nsUnzip v1.1 (current version) Test or extract files from a ZIP archive
ShellLink v1.2 (current version) Reads/Changes shell link (.lnk) files
SimpleSC v1.29 (current version) NSIS Simple Service Plugin
UAC v0.0.11d (not on NSIS wiki) Helps work around UAC problems on Vista/Windows 7
untgz v1.0.17 (current version) Extracts files from a simple tarball

Version information is not usually included in these DLLs so it can be difficult to identify a particular DLL.

Some plugins are rarely updated, others get frequent updates; this means that some plugins come in zip files which no longer match the current folder structure using by the NSIS compiler package.

To make it easier to build the installer and NSIS-based utilities for POPFile these extra plugins (and their associated documentation etc) have been repackaged as a single zip file ( 1.1.1-repackaged-plugins.zip) which can simply be unzipped into the main NSIS compiler folder (normally C:\Program Files\NSIS). A complete set of the original plugin zip files is also available ( 1.1.1-original-plugins.zip) as some of the plugins are not the most recent versions and are therefore harder to find.

FIXME Prepare 'original' and 'repackaged' collections of NSIS plugins used for the POPFile 1.1.2 installer

Obtain the source files from Subversion

The source code for POPFile, the POPFile installer and the NSIS-based POPFile utilities is stored in the POPFile project's Subversion (SVN) repository. This repository holds more than just the files used for the current release; it also holds the files used for all previous releases and the latest developments towards future releases. For further details see How to obtain the source code.

These notes will describe how to build a Windows installer for the current 1.x development version of POPFile.

Although the location of the source files used to build the installer is not hard-coded, the relative position of the source files is important. It is recommended that a new directory is created to hold the source files obtained from the Subversion repository and some of the additional utility software required to build the installer.

These notes will use “C:\Build\POPFile” as the build directory but a different location can be used if you prefer.

The first step is to retrieve (or “checkout”) the necessary source files from the project's Subversion repository. The easiest way to do this is by using a Subversion client, as mentioned in the How to obtain the source code page.

Use a Subversion client to checkout the current POPFile 1.x development program source files:

URL svn://getpopfile.org/branches/b0_22_2/engine
Destination directory C:\Build\POPFile\engine

Use a Subversion client to checkout the current POPFile 1.x development installer and other source files:

URL svn://getpopfile.org/branches/b0_22_2/windows
Destination directory C:\Build\POPFile\windows

Note that both of these checkout operations will actually checkout more files than are needed; this approach was chosen to avoid the need for multiple checkout commands.

Kakasi Package

POPFile uses a naïve Bayes algorithm to classify email. In other words, POPFile uses statistics to track which words are likely to appear in which messages. Japanese words are not separated by spaces (the English equivalent would look like Japanesewordsarenotseparatedbyspaces). This makes it harder for POPFile to classify messages. Therefore POPFile uses a special parser to split Japanese text into words to allow the text to be analysed properly. The 1.0.0 (or later) release offers a choice of three parsers (Kakasi, MeCab and internal). Prior releases of POPFile only supported the Kakasi parser.

The Windows installer includes all of the files needed for the Kakasi and internal parsers. There are two parts to the Kakasi parser: the Perl interface (Text::Kakasi) and the Kakasi package (some binary files and Japanese dictionary files). PPM can be used to install the Perl interface for Kakasi but the Windows version of the Kakasi package has to be downloaded separately.

Further information about the Kakasi package is available from the KAKASI (Kanji Kana Simple Inverter) site.

The current 32-bit Windows version of the Kakasi package is v2.3.4.

Create a directory called “kakasi_package” in the “C:\Build\POPFile\windows” directory.

The Kakasi package is distributed as a ZIP file which contains several folders (bin, doc, include, lib and share) all of which are under a top level folder called kakasi. Unzip the Kakasi package into the new C:\Build\POPFile\windows\kakasi_package directory.

The C:\Build\POPFile\windows\kakasi_package directory should now contain a directory called kakasi which contains the files and sub-directories for the Kakasi package.

SQLite command-line utilities

POPFile uses an SQLite database to hold essential information, such as the corpus used to classify messages. The SQLite library includes a simple command-line utility that allows the user to manually enter and execute SQL commands against an SQLite database. This utility can also be used to check the integrity of the database and defragment it.

POPFile currently uses SQLite 3.x format databases which are not compatible with the SQLite 2.x format used by some earlier versions of POPFile. In addition to the SQLite 3.x format command-line utility (sqlite3.exe) the installer also installs the SQLite 2.x format utility (sqlite.exe). Both utilities are available from the official SQLite website.

The installer usually includes SQLite command-line utilities which match the library versions used by the DBD::SQLite and DBD::SQLite2 modules it installs.

Module Name Version Library Version Download link for SQLite command-line utility
DBD::SQLite 1.31 SQLite 3.7.2 http://www.sqlite.org/sqlite-3_7_2.zip
DBD::SQLite2 0.33 SQLite 2.8.15 http://www.sqlite.org/sqlite-2_8_17.zip

Note: DBD::SQLite2 v0.33 is the most recent SQLite 2.x format module and v2.8.17 is the most recent SQLite 2.x format command-line utility.

Extract the “sqlite.exe” and “sqlite3.exe” files from the appropriate zip files and store them in the “C:\Build\POPFile\windows” directory.

Create the missing files

The release notes are not stored in the project's Subversion repository. The release notes are plain text files and should be stored in the C:\Build\POPFile\engine directory.

Use a text editor to create a plain text file called v1.1.2.change (for the English release notes) and another called v1.1.2.change.nihongo (for the Japanese release notes). Although the Windows installer can be built if the Japanese file is missing the cross-platform build will fail with an error message if the file cannot be found (so if no Japanese text is available just use a re-named copy of the v1.1.2.change file).

If the files checked out from Subversion are marked with a release number other than 1.1.2 then change the filenames of these two files to match (i.e. v1.1.2.change and v1.1.2.change.nihongo)

Build the Perl-based programs (optional)

FIXME Either build the six popfile*.exe files (may need to edit the engine\vars.mak file) or use the versions available from Subversion (or a suitable official release?)

Build the Windows installer

FIXME Add instructions here (use RC=“-SVN” option?)

make winexe
make xplat
make windows
make winexe
make xplat RC="-RC1"
make windows RC="-RC1"

make winexe uses the PDK to build the six Perl-based programs.

make xplat generates the cross-platform zip file (and I think it also generates a file which is required when building the Windows installer).

make windows build the Windows installer.

1)
If access to older versions (or builds for other platforms) is required an ActiveState Business Edition license has to be purchased.
2)
If ActivePerl is not installed in the default location you need to change the path specified in the “!define C_PERL_DIR” line of code in installer.nsi. For example, in the source for the POPFile 1.1.1 release edit line 327 of 'installer.nsi'
3)
The dumpstate plugin is not used by the installer or any of the NSIS-based utilities for POPFile. It is listed here as it is a very useful debugging aid.
 
devel/windowsinstaller.txt · Last modified: 2011/09/18 19:58 by 127.0.0.1

Should you find anything in the documentation that is incomplete, unclear, outdated or just plain wrong, please let us know and leave a note in the Documentation Forum.

Recent changes RSS feed Donate Driven by DokuWiki
The content of this wiki is protected by the GNU Fee Documentation License