Rattail File Monitor

The Rattail File Monitor application is a generic service which can monitor an arbitrary number of folders for incoming files, and perform an arbitrary number of actions on those files when they appear. It is implemented as a daemon on Linux, and a proper service on Microsoft Windows.

Installation

The basic Rattail File Monitor requires only the rattail package. (See Installation for details.)

Depending on which action(s) you intend the monitor to perform, you may need additional packages.

Registering the Windows Service

In addition to installing the Rattail package(s) for the file monitor, on Windows you must additionally register the service before it will run. This is simple enough; you may do so with this command:

C:\> rattail filemon install

If you wish the service to start automatically when the machine boots (as is typically the case), there is a convenient way to do this at registration time. Simply alter the command like so:

C:\> rattail filemon install --auto-start

Note that this command does not actually start the service; it merely registers it with a startup type of "automatic." (See also Automatic Startup below.)

Upgrading

To ensure a smooth upgrade of the file monitor service, the following sequence should be used:

C:\> rattail filemon stop
C:\> rattail filemon uninstall
C:\> easy_install --upgrade rattail
C:\> rattail filemon install --auto-start
C:\> rattail filemon start

Configuration

The file monitor requires a standard configuration file. The file must include one or more "monitor profiles" which will determine which folders are monitored, and which actions to take when incoming files arrive. In addition the configuration must specify which of the available "profiles" are actually to be used.

An example will help to illustrate. Since the file monitor has been tested primarily on Windows, we'll assume that environment. Suppose we want to watch a test folder on the desktop, C:\test\Incoming. When a file arrives we wish simply to move it to another test folder, C:\test\Outgoing. The following configuration would accomplish this:

[rattail.filemon]
monitored = test
test.dirs = [r'C:\test\Incoming']
test.actions = [('shutil:move', r'C:\test\Outgoing')]

Now to explain the contents of the example config:

First of all the monitored setting must be a comma-delimited list of profile names. In our example we had only one profile, test.

For each profile, two settings must also appear: one for the folder path(s) and another for the action(s) to take.

A profile's dirs setting must be valid Python code which evaluates to a sequence of strings, each of which must be a valid folder path. Each path contained is expected to already exist, and will be monitored for incoming files. In our example, the test.dirs setting defined only one "incoming" folder path.

A profile's actions setting must also be valid Python code, although the meaning is a little more complicated. It still must evaluate to a sequence, however each element in the sequence may either be a string or a tuple. Let's elaborate...

If the action element is a string, then it is taken to be a Python "spec" (dotted module path followed by a colon (:) followed by an importable attribute name). This spec will be loaded dynamically and is furthermore expected to be a Python callable. When a new file arrives, the callable will be called and passed the incoming file path as its first positional argument.

If the action element is a tuple, then the first element of the tuple must be a spec, and any remaining elements of the tuple will be passed as additional positional arguments (incoming file path will still be the first) to the callable which was loaded from the spec.

In our example we defined only a single action, with the tuple ('shutil:move', r'C:\test\Outgoing'). Therefore the move callable (function) will be imported from the shutil module. When a file arrives, move will be called with the incoming file path as the first argument, and rC:\test\Outgoing as the second. In other words assuming a new file of C:\test\Incoming\new.txt, the equivalent of the following will occur:

from shutil import move
move(r'C:\test\Incoming\new.txt', r'C:\test\Outgoing')

Now for a slightly more complete example config:

[rattail.filemon]
monitored = test1, test3

test1.dirs = [r'C:\test\Incoming1']
test1.actions = [
    ('shutil:copy', r'C:\test\Outgoing'),
    'os:remove',
    ]

test2.dirs = [r'C:\test\Incoming2']
test2.actions = ['os:remove']

test3.dirs = [r'C:\test\Incoming3', r'C:\test\Incoming4']
test3.dirs = ['os:remove']

So, the test1 profile will do basically the same as before, only the watched folder is named slightly differently. It also will perform two sequential actions, although the end result should be the same (i.e. copy then delete, instead of simply move).

The test3 profile will watch two different folders, but when files arrive within either of them they will simply be deleted.

Note that the test2 profile will not be used even though it is defined, because it is not included in the monitored setting.

Starting the Service

Even if you configured the service to auto-start on boot, you'll still need to start it manually the first time (unless you reboot). Do so with the command:

$ rattail filemon start

Notes

Linux

If you wish the daemon to start when the computer boots, you should add something like the following to your crontab:

@reboot /usr/local/bin/rattail --config=/usr/local/etc/rattail/filemon.conf filemon start

Note of course that this is just an example; the location of the rattail script and your config file may vary.

Microsoft Windows

On Windows, the file monitor is implemented as a proper Windows service (via PythonService.exe, from the Python for Windows Extensions). However there are a few things to be aware of:

Config File Location

There really is no way (currently) to pass command line arguments to the service itself, i.e. when it is being started. Therefore your config file must exist at one of the default file locations; you cannot override this in any practical way.

Automatic Startup

You more than likely will want to set the "Startup type" to "Automatic" so that the service starts when the computer is rebooted. However on Windows 7, it may be wiser still to set this to "Automatic (Delayed Start)" instead. The specific reason(s) for this are not entirely known. It may only affect those whose configuration requires accessing shared network paths (e.g. a network path is to be monitored, or a "chained" config file on the network is to be included).

If you edit the startup type manually within the Windows Services user interface, then it will be up to you to make the appropriate changes. However if you specify --auto-start when registering the service initially (see Registering the Windows Service), the "(Delayed Start)" type will be used automatically.

Exception Handling

There seems to be an issue with PythonService.exe and overriding the system-wide exception hook (sys.excepthook). In particular, it seems to do no good. My assumption is that PythonService.exe is serving the role of Python interpreter for the hosted service, and its implementation is slightly "faulty" in that it does not honor sys.excepthook.

This is unfortunate, as attempting to trigger email notifications for unhandled exceptions will not work. However since an unhandled exception is almost certainly to be caused from attempting to perform a configured action on an incoming file, that attempt is wrapped in a try/catch block and if it fails, an email is explicitly sent. The exception will not be re-raised; however any subsequent actions configured to be performed for the incoming file will be skipped.