<> = Deployment Framework Technical Overview = This page tries to explain some of the more technical aspects of how the deployment framework is structured, and how it expects to be used etc. See also the [[Deployment/Overview|General Overview]]. == The `fabfile.py` Script == Before describing any of the Rattail deployment specifics, it is worth noting that this system relies 100% upon [[http://www.fabfile.org/|Fabric]]. As such, everything starts with the `fabfile.py` script. The good news is that (with minor exceptions outlined below) the `fabfile.py` script should be a "normal" Fabric script. So we won't go over how Fabric scripts work here; you're expected to know something of that already. In general these docs always assume certain tasks will be present within a Fabric script: * `bootstrap_all` - Install/upgrade/configure all software such that the server is fully operational. * `production_sync` - Perform a full data sync (as necessary) from the "current" (live) production server, onto the target server. In practice you'd usually have more tasks than that in a given script; it depends on your needs. I typically split the bootstrap process into multiple components, e.g. a `bootstrap_system` and `bootstrap_rattail` and maybe `bootstrap_samba` etc. Then the `bootstrap_all` task would just call all the other ones. It should be noted that the `production_sync` task in particular is not yet "finished" in its design. It is mentioned here because the goal is to have a single task to accomplish the full data sync, i.e. backup/restore databases and rsync of relevant file paths. But the details are not yet as polished as the author would like; experimentation is underway as of this writing. == Fabric API Functions == Perhaps the most "basic" piece of the Rattail deployment framework is the [[https://rattailproject.org/trac/browser/rattail/rattail/fablib|rattail.fablib]] subpackage, which contains various functions designed for use with Fabric scripts. Some modules are dedicated to certain software systems (e.g. `rattail.fablib.postgresql`) while others may be more generic. The [[http://docs.fabfile.org/en/#api-documentation|Fabric API]] itself is quite good for basic server maintenance; in general the Rattail API only adds yet more convenience but doesn't do anything very revolutionary. For a simple example, consider this Fabric code: {{{#!highlight python numbers=disable from fabric.api import task, sudo @task def install_pg(): sudo('apt-get install postgresql') }}} Since `apt-get install` is something you'll probably be doing a lot of, there is a shortcut: {{{#!highlight python numbers=disable from fabric.api import task from rattail.fablib import apt @task def install_pg(): apt.install('postgresql') }}} The motivation behind these shortcuts is two-fold. On the one hand a shortcut is (or should be) simpler to read, but it also allows "best practices" to be consolidated and shared. For instance the `apt()` function above takes extra precautions to ensure that it happens in a fully-automated way, and does not stop to prompt the user for anything. Currently the best way to see what's available from the `rattail.fablib` API is to look through the source code. There really isn't much there, in fact. However one function in particular is among the most useful and deserves some explanation, next. === The `deploy()` Function === This is not a "normal" function per se; in fact there is not actually a `deploy()` function within the `rattail.fablib` package. However, a typical server bundle will make frequent use of a `deploy()` function. So, what's that about? First, the why: A common need in server maintenance is to push a config file to the server, overwriting any which may have previously existed there. In many cases this file may be considered "static" and is committed to the source repo in the same state as it will exist on the server (i.e. same content). However you may need certain config values within the file to be determined dynamically instead. In this case you'll need to commit only a "template" to the repo, and during deployment this template will be merged with dynamic values, and the ''result'' should be pushed to your target server. There also is the question of how to (most easily) indicate within the server bundle code, where exactly your local copy of the file resides. As a general rule, it's probably most convenient for the static/dynamic config files to reside within the bundle itself, although there are exceptions (e.g. a common `.gitconfig` that you wish to install to all servers). These two reasons are why there is a special `deploy()` function. It provides a single, flexible way to push files to the server whether they are "static" or dynamic. And, it provides a way to essentially keep local file paths "relative" when specified in code. So let's get on to how to use it: Consider the following basic server bundle file structure: {{{ |-- fabfile.py `-- deploy `-- rattail |-- rattail.conf.template `-- filemon.conf }}} This structure is not complete; see the [[#Server_Bundle|Server Bundle]] section below for more about what the structure should actually look like. Here we're only really interested in what's in the 'deploy' folder, which happens to be two config files under a 'rattail' subfolder. You'll notice that one of the file names ends with '.template' - that is a special suffix which will cause the file to be merged with the running Fabric environment before being pushed to the server, instead of being pushed as-is (i.e. as a "static" file). Within the `fabfile.py` script then, might be the following code: {{{#!highlight python numbers=disable from fabric.api import task from rattail.fablib import make_deploy, mkdir deploy = make_deploy(__file__) @task def configure_rattail(): mkdir('/etc/rattail') deploy('rattail/rattail.conf.template', '/etc/rattail/rattail.conf') deploy('rattail/filemon.conf', '/etc/rattail/filemon.conf') }}} Here you can see where the `deploy()` function actually comes from: It is created dynamically with the `fabfile.py` path as its "context". This is how we can then specify relative file paths when calling `deploy()` within the task. In other words, the `deploy()` function itself considers its "working directory" to be the 'deploy' folder which exists alongside the 'fabfile.py' script. It is important to note that this much is effectively hard-coded, meaning if you changed the 'deploy' folder name to say, 'server-files', then you would break the magic. The other piece of magic here involves the static vs. dynamic behavior. In the case of `filemon.conf` it will upload the local file as-is to the server, but for `rattail.conf` it will first generate a "final version" of the file which it then pushes to the server. This is merely a convenience wrapper of sorts, under the hood it's calling either the Fabric [[http://docs.fabfile.org/en/1.10/api/core/operations.html#fabric.operations.put|put()]] or [[http://docs.fabfile.org/en/1.10/api/contrib/files.html#fabric.contrib.files.upload_template|upload_template()]] function. == The `fabenv.py` Script == Another distinguishing feature of a Rattail-style server bundle is the presence of a `fabenv.py` script. This is a simple Python module whose purpose is only to set local/custom attributes in the Fabric environment, for use in generating dynamic files to be pushed to the server, and in some cases conditional logic etc. Such a file might look like: {{{#!highlight python numbers=disable from fabric.api import env env.password_postgresql_rattail = 'somepassword' env.password_mssql_locsms = 'anotherpassword' }}} This file must be imported by the `fabfile.py` script explicitly, there is no magic to avoid that. To further flesh out the 'deploy' example from earlier: {{{#!highlight python numbers=disable from fabric.api import task from rattail.fablib import make_deploy, mkdir # Note: This line is the only difference. import fabenv deploy = make_deploy(__file__) @task def configure_rattail(): mkdir('/etc/rattail') deploy('rattail/rattail.conf.template', '/etc/rattail/rattail.conf') deploy('rattail/filemon.conf', '/etc/rattail/filemon.conf') }}} As is implied with this syntax, the `fabenv.py` script must exist alongside the `fabfile.py` script. This also means that you must run the Fabric script from within its own directory. Now that the `fabenv.py` script is being imported, the `rattail.conf.template` file is free to leverage any of its values by name, e.g.: {{{#!highlight ini numbers=disable [rattail.db] default.url = postgresql://rattail:%(password_postgresql_rattail)s@localhost/rattail }}} Given the nature of the contents of the `fabenv.py` script, it is important that it never be committed to the source repo. In fact the typical repo will have a VCS exclusion rule so that all files with this name are permanently ignored. However it also is good practice to maintain a "template" for the `fabenv.py` script, so that it's at least easier to quickly see which passwords etc. you do need to set when preparing the bundle for real use. This template ''should'' then be committed to the repo; it is usually named `fabenv.py.dist` by convention. Its contents are the same as `fabenv.py`, except the actual passwords are blanked out etc. == Server Bundle == The term "bundle" just refers to the collection of logic and data required to install and configure a server. In practice it is expected that this collection resides within a single folder (albeit with at least one subfolder, 'deploy'). One sort of exception to this rule is that some logic which is used by the server bundle's code, may reside in other Python packages (e.g. `rattail`). But in all cases the bundle itself is the "final authority" for the configuration of its associated server. Note then that a key concept here is the one-to-one association of a server bundle. There is one bundle per server, and one server per bundle, always. The "bundle" is also just a single term which "puts it all together" as far as the various concepts described above. Meaning, within a bundle you will have a `fabfile.py` and a `fabenv.py`, and you will call various Fabric API functions, and have a `deploy` subfolder with various config files, and you'll call the `deploy()` function to push them to the server, etc. Some of the file structure was outlined above, but here is a complete (minimal) example: {{{ /home/lance/src/myfab/servers/host/ |-- deploy | |-- ssh | | `-- id_rsa | `-- rattail | |-- rattail.conf.template | `-- filemon.conf |-- fabenv.py |-- fabenv.py.dist `-- fabfile.py }}} That's it! Not much more than you've already seen, really. Again, the `fabenv.py` (if present) is a local-only file and never committed to the repo. The `id_rsa` file above is meant to show an example of a private/secure file which you also would not wish to commit to the repo. However you can see that as long as the file ''exists'' there locally, you would be able to call `deploy()` on it just like any other file. In practice then, such a file should also be explicitly ignored by VCS to avoid accidental commits.