The Big Tour

Here we try to give a "full tour" of the Rattail Project in one page. It's intentionally brief in many areas but plenty wordy (and technical) still; it aims to at least touch on "all" the birds-eye-view details.

Really this page tries to answer the question for the reader - who is assumed to be a developer or sysadmin! - "Should I even be interested in Rattail?" By the time you've read this page you should have the answer to that.

In a nutshell, Rattail tries to offer a "basic and unobtrusive, yet opinionated and convenient" Swiss Army Knife / toolkit for building custom back-end / automation software for integrating with other systems and interacting with that data. Retailers were the assumed market for this software but in practice it can be used for many other purposes / organizations as well.

To some extent each of the primary aspects of the Rattail Project (below) can be used independently, so feel free to start wherever you like in practice.

Remember, the word "Poser" here really means "Custom Project" - i.e. your custom project - so replace that name mentally with whatever you like, i.e. whatever you want to name your custom project.

Requirements

Most all of the Rattail Project source code is written in Python which presumably is cross-platform, although all development is done exclusively on/for Linux.

Basically the only thing which runs on Windows in the wild is the Rattail Filemon service - which does run quite nicely, but is a different beast somewhat than the equivalent Filemon daemon on Linux.

Another point for Linux, strike against Windows, is that the "config management" approach used by Rattail (see further below) is very capable of managing Linux machines but is unable to manage Windows machines.

Therefore in practice, Linux feels like a requirement for the most part, and should (hopefully) be embraced as such. Debian and Ubuntu have each been used for this quite extensively.

Beyond that, if you intend to use a Poser database, it is assumed to exist in PostgreSQL. Again it's possible that other database engines would work, but that would likely require a tweak here and there, and in practice you should just use PostgreSQL.

Configuration

Here are a few things worth knowing about how a Poser/Rattail app is configured.

Files

The primary method of config is via so-called INI files, which contain "sections" and "option / value" pairs.

Such config files may "inherit" from each other, so that your app may require one config file but then several others are indirectly loaded due to this inheritance. The inherited "values" will be overridden by any values present in a "descendent" file, e.g. the one your app uses.

It also is possible for the app to specify multiple config files instead of just the one; in this case each of these will still inherit (as defined in each file).

DB Settings

If there is a Poser database present for the app, then it usually is helpful to allow config to be read from the Settings table in this DB, as well as the normal files. This is not required though and is off by default.

The big win here is, you can then edit settings in the database directly from the Poser web app, and they take effect across the app "immediately" - whereas if you edit a config file, you must restart the app for it to take effect. This allows for (more convenient) so-called feature flags, meaning you launch/upgrade the app with support for a certain new feature hidden behind a setting which you then "turn on" whenever you're ready.

Note that some config values will never be read from DB and only can come from the files (e.g. database connection parameters). Generally speaking any "sensitive" values should probably only be stored in the files (with appropriate security).

Database

It often is useful to maintain one or more Poser databases, to facilitate automation or other organization-defined workflow. Note that the web app requires a database.

Schema Definition

Rattail provides a "core" ("common") database schema, with tables for users, products, customers, etc. Each custom Poser app is free to define "extensions" to this schema, to track things not represented by the core schema. (It's generally expected that most Poser apps would do so.) These schema definitions are made using the SQLAlchemy ORM library/syntax.

Note that whereas you can pick and choose which parts of Rattail software you want to run, the schema is "all or nothing" - meaning each DB built using the Rattail framework will effectively contain this full "core + extensions" schema. In practice that's no big deal - just ignore any tables you don't need within app logic, as the core schema is sure to define some you don't need!

Location / Purpose

The assumption is that generally each Poser app will require one "core" Rattail database, which will live in PostgreSQL. At the very least, this DB would be needed to store user accounts and roles/permissions for the Poser web app, if that is deployed. Beyond that, it's often useful to treat this DB as a "pseudo-authority" - meaning, with the right setup you can expect this DB to always contain the "latest" authoritative data (assuming datasync etc.) in which case you can then develop further reports and app features based on this "custom but very accurate" DB. Additionally it can be useful to treat this DB as a "workspace" for moving data between other systems but with extra processing / validation steps etc.

In some cases (e.g. minimal integration) no such database is needed; in other cases (e.g. multi-store) there may be multiple Poser apps with each requiring its own database.

Schema Upgrades

Regardless of how many Poser databases must exist, they should generally be kept up-to-date such that the schema of each is the same. Among other reasons, this is because the datasync daemon will usually try to sync all fields for a record, in which case those field definitions must match between app nodes. Alembic is the tool used to track migrations and keep schema for each database upgraded.

Data

As for how data makes its way into this database, that will vary of course but by and large will be via "integration" (with other systems) and/or "web app" (aka. integrating with users) - both of which are discussed further below.

Note that if there are multiple Poser databases, then often a new data change in one node, will be propagated to all other nodes, typically by way of a "host" (master) node. When applicable, this would be handled via datasync.

Data Versions

Poser's database includes support for "data versioning" via SQLAlchemy-Continuum. The versioning is a slightly tricky feature, in that it must be turned on or off at app startup, and the versioning can't be turned back off/on without restarting the app. In practice that's not normally too big a deal, and there are workarounds available when needed. (The main issue is that some importers and batches can be slow to process when versioning is enabled in the web app.)

But enough of that, now for the good news. You are in fact encouraged to turn on versioning wherever you like, it's a pretty solid feature. The idea is that for a given "versioned" table, a second table is created to match it basically, and this table will include a new "version" record for each time the "main" record is changed in any way. With versioning turned on, this will happen automatically; with it off, it won't.

The version history of a given "primary" record then can be made available within the Poser web app. Note that due to technical issues with how this feature works (again, tricky), this version history can only be displayed in the web UI if the versioning feature is currently turned ON.

Other Databases

Thus far we have only described the "core" DB schema provided by Rattail. But in fact it offers a couple more, for specific purposes:

Trainwreck

Rattail defines a basic / common schema for POS Transactions, for what it calls a "Trainwreck" database. Again you're free to extend this schema; the idea is that a Trainwreck DB would be robust enough to store all "pertinent" info about transactions, in a POS-neutral way.

You can read more about this on the Trainwreck page.

TempMon

In a separate rattail-tempmon package, Rattail defines a so-called "TempMon" DB schema. This one is assumed to be sufficient for the intended use case, although could likely be extended in the same way as for the "core" Rattail/Poser DB if necessary.

The use case for this, is to monitor temperatures for one or more "probes" which are recording temperature data for one or more "appliances". This is a pretty cool feature (sends email alerts when things get too warm etc.) but unfortunately not very well documented at this time. It's assumed that each device (with probes) is a Raspberry Pi and is running the rattail tempmon-client "daemon" software, writing to a central DB. (And there is a rattail tempmon-server daemon running on the central server, which does the actual "monitoring" of temperatures.)

Integrations

By "integration" we mean the providing and/or receiving of data and/or commands between the Poser/Rattail system and an arbitrary number of other systems.

Next we'll expand on that definition a bit, then go over the technical "styles" of integration provided by Rattail.

Concepts

Let's say you're a retailer with a POS (point of sale) system, which serves as a defacto CRM (customer relations management) system also. In other words "all your customer records" live in the POS. Let's say you also have an online email campaign (mailing list) to which you would like to push the email addresses of new customers as they are added to your POS. This is one of many "integration" scenarios which may be handled via Poser/Rattail.

Continuing with this example, you have some options. It's possible that your POS already has a plugin which could provide the solution. Or, you could use Poser to dynamically read customer data from POS (presumably via ODBC/SQL), and from the email campaign also (presumably via web API), then push any "missing" addresses to the campaign. It may be helpful to keep a "local cache" of data from one or both systems, within a Poser database. This can make for a more efficient app, albeit with more moving parts. But then you could also run a Poser web app to give a read-only view of "all" the data in one place.

Pretty much all integrations are like this in some sense. Poser would need to have a way to read and/or write data from/to the given system(s). When and how it does so would ultimately be up to Poser business logic etc. The Poser web app (and even the Poser database) is optional until you want a feature which would require it.

Note that in this example, we mentioned your POS may already have a plugin to do the job. To expand on that point, the Poser/Rattail approach is not "novel" per se - most of what it does can be accomplished in some other way, easily enough. Sometimes this "other way" would resemble a Poser approach in many ways, but using a different tech stack etc. Especially when only "one" integration feature is considered at a time, there may be no compelling reason to use the Poser/Rattail approach. But what the latter does provide, is a common framework which is capable of tying all integration efforts into one cohesive whole.

Data Import / Export

The "meat and potatoes" of integrations are really the import/export of data between systems. Rattail provides an "importer framework" which is meant to do most of the heavy lifting, and in many cases a new importer need do little more than define which fields it supports, and how to obtain a value for each, for a given object/record.

Note that from this framework's perspective, the "import vs. export" distinction is largely meaningless; it always assumes (essentially) a "Source -> Target" scenario which may be thought of as "Target imports from Source". You may declare a given importer to really be an "exporter" in which case that would be thought of as "Source exports to Target" instead - but everything about how it actually works, would be the same. Declaring an importer to be of "export" persuasion is only for the user interface, so it makes more sense to the humans.

The general idea of the importer framework, is that for a given pair of systems, and data flow direction (e.g. POS -> Poser), there will be one "import handler" defined. This is responsible for, among other things, establishing a database transaction for either/both of the systems, and committing it (or rolling back, if dry run) at the end. The handler also will be configured with one or more "importers" - each of which is responsible for a "model" and how to map data for a record from one system, to the other. So the "handler" is for the DB and each "importer" corresponds to a table, conceptually. This is a simplistic overview and many importers in the wild bend these definitions somewhat.

The framework provides a fairly robust command line interface for each import handler defined. The commands are optimized for efficiency and allow for toggling the create/update/delete support, as well as restricting which fields are updated in a given run, etc. Such commands may be scheduled via cron and voilà, you have (poor man's / pseudo) data sync between systems. It also has a "warnings" mode which basically means "no changes are expected, so send email report if any are found" - as well as a "dry run" mode which can be useful when testing ad-hoc commands.

Generally importer "jobs" (runs) are expected to be either automated (via cron) or else ran manually via command line. In some cases the web app may support running certain types of importer jobs, as needed. It is even possible to convert a new/incoming importer job to a "batch" so its contents may be fully inspected within the web app prior to "execution".

Real-Time Data Sync

If the poor man's data sync (i.e. running importers via cron) isn't good enough, you might want to go with the Rattail "DataSync" proper. This is a daemon with multiple threads running, some of which will "watch" for new data changes which appear in a given system, while others will then "consume" the changes by pushing updates to some other system.

This "datasync" daemon is meant to leverage the "importer" framework as much as possible. So for instance, one watcher thread will notice say, a new Customer record has appeared in the POS system. Then a consumer thread will create the appropriate importer, and tell it to "import" the new customer record (which might say, push a new email address to the mailing list). In this way you're able to keep the logic as "DRY" as possible. (Adding datasync support rarely requires much new code although does assume importers contain most of the logic.)

However this datasync daemon does require a Poser database - because it uses a table within it as a "queue" for incoming changes which await processing.

While your Poser app may have multiple import handlers defined, you will need only one datasync daemon per app node (if indeed, you need one at all). Multiple importers can be "handled" within a single datasync daemon; it merely creates additional threads as configured.

Data Batches

The "batch" concept is a bit like an importer, but the real idea for a batch is that certain data will be brought (from one or more systems) into a "workspace / temp" table in the Poser DB, and then some user(s) will review and perhaps modify this temp data, and ultimately "execute" the batch - which would update Poser and/or other systems as appropriate.

So, while you can usually create a batch "from" an importer job (run), and thereby delay "execution" for the importer job (pending user input), that is not the typical use case. Most batches are "purpose-built" in some way, and serve a different function than normal importers would. Some common examples are:

• Purchasing (create/submit new purchase order to vendor)
• Receiving (process receiving for vendor purchase order / invoice)
• Inventory (process counts coming from mobile/handheld batch)
• Labels (print shelf tags for set of products)
• Product Import (e.g. custom "pseudo-importer" of new products from spreadsheet)

After saying most batches are "not" importers, I still threw in that last example! To highlight the difference, a "true" importer in Rattail-speak is meant to be part of the automation, whereas a batch by definition is meant to require user input (and therefore not "automated" - at least not fully). But it's often the case that someone in charge of adding products to the system, is very Excel-savvy and wants a tool to do so directly from some spreadsheet of their own design (for instance). It can then be helpful to bring this data in via batch instead of proper importer, so the user can be more involved and confident in the details. And, while "any" importer can be used to create a "dynamic" batch, a purpose-built batch type will provide a better experience. (And they're not that hard!)

Really you can use batches to do just about anything, where you want the user to get to preview the entire job before hitting the execute button. Some batches are "immediately" populated with data when first made, e.g. a Receiving batch might be populated with contents of a PO or invoice. Other batches are slowly populated over time, e.g. Purchasing where a user scans shelf tags while building the order. And yet, because it's a free country, you can automate a batch if you want - e.g. automatically create/populate/execute it with no user intervention.

Since batches generally require user input, they also will generally require a Poser web app.

Note that while we refer to a "temp table" here, really each batch type has its own set of persistent tables, so each "batch" is really just a set of records within those tables. The default behavior is for these records to "stick around" after the batch has been executed, which may be useful as an audit trail of sorts. Each batch may be executed only once, after which it is frozen in time. (Who created and executed the batch is also recorded.) You may want to purge old batch records occasionally, if any tables get heavy use.

File Monitor

Rattail's "FileMon" is another daemon, somewhat like the "DataSync" daemon, but instead of watching databases and syncing those changes, the filemon...monitors files. And due to the nature of files, this isn't always about the "data" per se, meaning sometimes this gets more into the "command" side of things.

The general idea though again is not unlike datasync, in that the filemon daemon has a bunch of threads running, some of which will "monitor" ("watch") a given folder for new files to appear, and other threads will then "process" these files in some way. In most cases the files are either deleted outright, or perhaps archived to another folder, after processing.

The filemon can be useful for keeping tabs on a POS system, if it's the type to generate new files in a configured location, for e.g. new transactions which occur at the register. Another use would be to actually send "commands" between Poser nodes, in a "resilient" manner. For instance, to print a set of labels for all stores, from a multi-store host node, the host web app can generate a file with all the details, and then effectively hand it off to the filemon for distributing around to the other nodes.

But in the end, anytime you need to watch a folder for new files, and process them, you would probably want to use filemon.

Web App

There is a lot to say about web apps, but we'll try to keep this brief...

Location / Purpose

These aspects are combined here because one usually influences the other. The typical setup at the very least will involve one "host" (i.e. "central") app node for Poser, which has its own database as well as running its own web app.

This "typical" web app has as its #1 objective, to provide a "view" of the data within its own database. It may also provide views of data within other databases from "external" (non-Poser) systems.

Sometimes also the app requires input from the user(s) and this is accomplished most conveniently with a web app.

The web app framework is very table-oriented, and generally speaking each "table" gets its own "master view" defined within the app. The "home page" for this master view will be the table itself, more or less. This is known as the "index" or "grid" view. Users may filter/sort this table as they attempt to locate pertinent records. The "master view" also provides CRUD views for the record, although which are actually supported will be up to the app (i.e. it can disable some as needed).

This web app typically runs on the same server/machine as the database. If the app generally must be on the local network, e.g. for sake of integrating with the POS system, then the web app will be local too. However in some cases the app/database can live "in the cloud" in which case that's where the web app will be also.

Security

Any access restrictions based on client IP or anything "fundamental" like that are assumed to be taken care of by your Apache (or similar) config, and not the job of Poser app logic for instance. However you can do whatever you like with that.

The main security features of Poser/Tailbone/Rattail are with regard to user authorization. User accounts are stored in the database, with passwords encrypted. Users belong to one or more Roles and then each Role is granted or denied access to each Permission (feature). A user's effective permissions then will depend on which roles they belong to.

Users who belong to the "Administrators" role are granted another special permission - they may "Become Root" which does sort of the same thing as within Linux shell; it grants full access to all app features until the user exits this mode.

User login/logout events are logged within the database but no other actions are performed based on this, and no other actions are logged necessarily. Apache or similar logs may offer more insight to app usage.

Desktop vs. Mobile

The web app framework definitely has its roots in the "desktop" world. It's rendered server-side and has traditionally used jQuery UI on the frontend, with no eye whatsoever to mobile/responsive concerns.

As of this writing a new desktop "theme" based on Buefy (and Vue.js) is almost complete. This will certainly improve the situation for the "default" web app in terms of responsiveness at least, although the assumption will remain that this particular web app will generally be accessed by desktop users. (Also it will continue to be server-rendered at the fundamental level, i.e. not a "single-page" app.)

However for some time now Tailbone has also supported a "mobile" web app which uses jQuery Mobile and which tries only to expose tools required by a mobile user. This mobile app would need a new "theme" (ahem, "rewrite") at least, based on Buefy also, although no effort has been made to that yet. The existing mobile app can be useful though, for e.g. product ordering, receiving, inventory and the like.

"Other" Web Apps

While the default web app framework is pretty handy - it's not the only way to go about that. Usually a given app node would include a "typical" web app, but maybe it also needs to expose a purpose-built web app, e.g. for a better user experience.

One common example would be a "customer accounts" online portal website. Your main app node might be on the local network, but you have this "customer accounts" node in the cloud. Data is synced between the two, and each node has a "typical" web app, but the one in the cloud rarely gets used and is there just for admin's sake. The cloud node instead would run a custom "accounts" web app in addition, which is very user-friendly for the customers. This custom web app must still interact directly with the cloud database but then datasync takes it from there.

App Emails

The primary concern for a Poser app is usually just, how/when/why to send emails out based on business logic and events. Most of this section is concerned with that side of things.

However in some cases you may need to detect email "bounces" and take action; that also is addressed at the end.

What Triggers Email

Lots of things really. Some common examples:

• "warnings" diff from an importer job/run
• "problem reports" - arbitrary sanity checks which can produce simple reports
• errors - basically anytime an unhandled exception occurs, email should ensue
• cron output - most jobs should emit no output, so emails are an exception here (ideally)

Those are sort of the "boring" examples though. Poser is free to send email whenever it likes, e.g. when a particular type of batch is executed, or anything else.

Note also that most of the examples above, would involve sending email to "staff" (and perhaps even just "admin") users, as opposed to the general public. The latter is supported also but can raise additional concerns for the implementation.

Types of Email

From the Rattail perspective there are fundamentally only two types of email: "native" and "other" - where "native" means email sent directly by the Rattail framework and "other" means email sent by any other means.

Most of the "What Triggers Email" list above would be of the "native" type but not all, e.g. cron output is usually not.

Python's logging module can be configured in such a way that anytime an "error" is logged, it can be sent out via email. Many apps will choose to do this as it can be quite convenient to just "log an error" instead of setting up a proper "native" email with template etc.

The advantage to "native" emails is that you can define a template and more flexibly configure sender, recipients, subject etc. Within the Rattail framework each native email is further assigned a particular "type" - e.g. "new customer added" or "big sales day" etc. This type has a "key" (e.g. new_customer_added) and the template/config for each email type will be named according to this key.

Native email types which have been fully defined / supported in Poser, can be edited within the web app. (Er, sender/subject etc. may be edited, but not the template.) Also each email may be "previewed" which shows HTML etc. with some sample data in-browser, or can send out a proper email preview.

Sending Mechanics

Usually the Poser app node will run on a Linux machine, in which case the assumption is that it also runs Postfix, and therefore the Poser app itself need only send mail by way of SMTP on localhost. Of course, Postfix may then be configured to use a relay etc., but again Poser itself should not be concerned about tricky sending config. In practice it should not even need to authenticate to localhost Postfix, for instance.

Poser can be configured to record Email Attempts within the database. If this is enabled, it will make new records with sender, recipient, subject etc. whenever "native" email is sent via the framework. Note that (as of this writing) it does not store the full message body, but just a few header values etc.

Email Bounces

Anytime you're sending email out to the general public, you may want to know when bounces are happening since you therefore may have invalid addresses on file.

To this end you can use Rattail's "Bouncer" daemon to periodically check an IMAP folder. It knows how to detect bounce messages and can then invoke whatever action(s) you define when a bounce is detected, e.g. internally mark a customer's email address as invalid.

Config Management

We're using the term Configuration Management here because it seems to be the accepted one. At any rate what we're concerned about is that all machines running software under our control, should be "configured" in a "managed" way, so that we can replace them more easily etc.

After all you will sleep much better knowing that if any machine needs replacing / "moving" etc. you can reconstruct all software environments with minimal fuss. Getting the data moved over is a slightly different matter but still can usually be pretty straightforward.

Rattail can't claim too much credit here, most of the heavy lifting is done by Fabric and we only provide a set of opinions and conveniences, basically. But it's a good framework for managing your servers if you don't already have one, e.g. Ansible which arguably is more popular. Fabric's approach is a bit different, so the resulting code is more like a script than a config file, generally speaking. It's still just Python though and quite readable.

The idea is that you maintain a local "control environment" e.g. on your laptop, which contains all Fabric-related code. From here you can run simple commands which will effectively SSH into a target machine, and run some command(s) there which will further install and configure all software. Various files may be held in the local repo which can be "deployed" (copied) to the target machine. Note that any "secrets" (e.g. passwords) should not be committed to the local code repo, but still must exist within the local working dir (e.g. within ignored files) for the sake of deployment to the target machine.

One caveat should be mentioned. Thus far the "target" machines have all been either Debian or Ubuntu, and what logic Rattail provides will assume a Debian derivative at least, although Fabric itself would be happy targeting any Linux machine I assume. But there is no support for targeting Windows with this method, that I know of. Furthermore the local "control environment" is also assumed to be Linux. It's possible this could be done from a Windows laptop but that is untested.