syncsource.py

OVERVIEW

The syncsource.py utility transfers groups of files between
computers. The primary use case is to enable developing LLVM project
software on one machine, transfer it efficiently to other machines ---
possibly of other architectures --- and test it there. syncsource.py
supports configurable, named source-to-destination mappings and has a
transfer agent plug-in architecture. The current distribution provides
an rsync-over-ssh transfer agent.

The primary benefits of using syncsource.py are:

* Provides a simple, reliable way to get a mirror copy of primary-
  machine files onto several different destinations without concern
  of compromising the patch during testing on different machines.

* Handles directory-mapping differences between two machines.  For
  LLDB, this is helpful when going between OS X and any other non-OS X
  target system.

EXAMPLE WORKFLOW

This utility was developed in the context of working on the LLDB
project.  Below we show the transfers we'd like to have happen,
and the configuration that supports it.

Workflow Example:

* Develop on OS X (primary machine)
* Test candidate changes on OS X.
* Test candidate changes on a Linux machine (machine-name: lldb-linux).
* Test candidate changes on a FreeBSD machine (machine-name: lldb-freebsd).
* Do check-ins from OS X machine.

Requirements:

* OS X machine requires the lldb source layout: lldb, lldb/llvm,
  lldb/llvm/tools/clang. Note this is different than the canonical
  llvm, llvm/tools/clang, llvm/tools/lldb layout that we'll want on
  the Linux and FreeBSD test machines.

* Linux machine requires the llvm, llvm/tools/clang and
  llvm/tools/lldb layout.

* FreeBSD machine requires the same layout as the llvm machine.

syncsource.py configuration in ~/.syncsourcerc:

# This is my configuration with a comment.  Configuration
# files are JSON-based.
{ "configurations": [
    # Here we have a collection of named configuration blocks.
    # Configuration blocks can chain back to a parent by name.
    {
        # Every block has a name so it can be referenced from
        # the command line or chained back to by a child block
        # for sharing.
        "name": "base_tot_settings",

        # This directive lists the "directory ids" that we'll care
        # about.  If your local repository has additional directories
        # for other projects that need to ride along, add them here.
        # For defaulting purposes, it makes sense to name the
        # directory IDs as the most likely name for the directory
        # itself.  For stock LLDB from top of tree, we generally only
        # care about lldb, llvm and clang.
        "dir_names": [ "llvm", "clang", "lldb" ],

        # This section describes where the source files live on
        # the primary machine.  There should always be a base_dir
        # entry, which indicates where in the local filesystem the
        # projects are rooted.  For each dir in dir_names, there
        # should be either:
        # 1. an entry named {dir-id}_dir (e.g. llvm_dir), which
        #    specifies the source directory for the given dir id
        #    relative to the base_dir entry, OR
        # 2. no entry, in which case the directory is assumed to
        #    be the same as {dir-id}.  In the example below, the
        #    base_dir-relative directory for the "lldb" dir-id is
        #    defaulted to being "lldb".  That's exactly what
        #    we need in an OS X-style lldb dir layout.
        "source": {
            "base_dir": "~/work/lldb-tot",
            "llvm_dir": "lldb/llvm",
            "clang_dir": "lldb/llvm/tools/clang"
        },

        # source_excludes covers any exclusions that:
        # * should be applied when copying files from the source
        # * should be excluded from deletion on the destination
        #
        # By default, ".git", ".svn" and ".pyc" are added to
        # all dir-id exclusions.  The default excludes can be
        # controlled by the syncsource.py --default-excludes
        # option.
        #
        # Below, I have transfer of the lldb dir skip everything
        # rooted at "/llvm" below the the lldb dir.  This is
        # because we want the source OS X lldb to move to
        # a destination of {some-dest-root}/llvm/tools/lldb, and
        # not have the OS-X-inverted llvm copy over with the lldb
        # transfer portion.  We'll see the complete picture of
        # how this works when we get to specifying destinations
        # later on in the config.
        #
        # We also exclude the "/build" and "/llvm-build" dir rooted in
        # the OS X-side sources.  The Xcode configuration on this
        # OS X machine will dump lldb builds in the /build directory
        # relative to the lldb dir, and it will build llvm+clang in
        # the /llvm-build dir relative to the lldb dir.
        #
        # Note the first forward slash in "/build" indicates to the
        # transfer agent that we only want to exclude the
        # ~/work/lldb-tot/lldb/build dir, not just any file or
        # directory named "build" somewhere underneath the lldb
        # directory.  Without the leading forward slash, any file
        # or directory called build anywhere underneath the lldb dir
        # will be excluded, which is definitely not what we want here.
        #
        # For the llvm dir, we do a source-side exclude for
        # "/tools/clang".  We manage the clang transfer as a separate
        # entity, so we don't want the movement of llvm to also move
        # clang.
        #
        # The llvm_dir exclusion of "/tools/lldb" is the first example
        # of an exclude targeting a requirement of the destination
        # side.  Normally the transfer agent will delete anything on
        # the destination that is not present on the source.  It is
        # trying to mirror, and ensure both sides have the same
        # content.  The source side of llvm on OS X does not have a
        # "/tools/lldb", so at first this exclude looks non-sensical.
        # But on the canonical destination layout, lldb lives in
        # {some-dest-root}/llvm/tools/lldb.  Without this exclude,
        # the transfer agent would blow away the tools/lldb directory
        # on the destination every time we transfer, and then have to
        # copy the lldb dir all over again.  For rsync+ssh, that
        # totally would defeat the huge transfer efficiencies gained
        # by using rsync in the first place.
        #
        # Note the overloading of both source and dest style excludes
        # ultimately comes from the rsync-style exclude mechanism.
        # If it wasn't for that, I would have separated source and
        # dest excludes out better.
        "source_excludes": {
            "lldb_dir": ["/llvm", "/build", "/llvm-build"],
            "llvm_dir": ["/tools/lldb", "/tools/clang"]
        }
    },

    # Top of tree public, common settings for all destinations.
    {
        # The name for this config block.
        "name": "common_tot",

        # Here is our first chaining back to a parent config block.
        # Any settings in "common_tot" not specified here are going
        # to be retrieved from the parent.
        "parent": "base_tot_settings",

        # The transfer agent class to use.  Right now, the only one
        # available is this one here that uses rsync over ssh.
        # If some other mechanism is needed to reach this destination,
        # it can be specified here in full [[package.]module.]class form.
        "transfer_class": "transfer.rsync.RsyncOverSsh",

        # Specifies the destination-root-relative directories.
        # Here our desination is rooted at:
        # {some-yet-to-be-specified-destination-root} + "base_dir".
        # In other words, each destination will have some kind of root
        # for all relative file placement.  We'll see those defined
        # later, as they can change per destination machine.
        # The block below describes the settings relative to that
        # destination root.
        #
        # As before, each dir-id used in this configuration is
        # expected to have either:
        # 1. an entry named {dir-id}_dir (e.g. llvm_dir), which
        #    specifies the destination directory for the given dir id
        #    relative to the dest_root+base_dir entries, OR
        # 2. no entry, in which case the directory is assumed to
        #    be the same as {dir-id}.  In the example below, the
        #    dest_root+base_dir-relative directory for the "llvm" dir-id is
        #    defaulted to being "llvm".  That's exactly what
        #    we need in a canonical llvm/clang/lldb setup on
        #    Linux/FreeBSD.
        #
        #    Note we see the untangling of the OS X lldb-centric
        #    directory structure to the canonical llvm,
        #    llvm/tools/clang, llvm/tools/lldb structure below.
        #    We are mapping lldb into a subdirectory of the llvm
        #    directory.
        #
        #    The transfer logic figures out which directories to copy
        #    first by finding the shortest destination absolute path
        #    and doing them in that order.  For our case, this ensures
        #    llvm is copied over before lldb or clang.
        "dest": {
            "base_dir": "work/mirror/git",
            "lldb_dir": "llvm/tools/lldb",
            "clang_dir": "llvm/tools/clang"
        }
    },

    # Describe the lldb-linux destination.  With this,
    # we're done with the mapping for transfer setup
    # for the lldb-linux box.  This configuration can
    # be used either by:
    # 1. having a parent "default" blockthat points to this one,
    #    which then gets used by default, or
    # 2. using the --configuration/-c CONFIG option to
    #    specify using this name on the syncsource.py command line.
    {
        "name": "lldb-linux"
        "parent": "common_tot",

        # The ssh block is understood by the rsync+ssh transfer
        # agent.  Other agents would probably require different
        # agent-specific details that they could read from
        # other blocks.
        "ssh": {
            # This specifies the host name (or IP address) as would
            # be used as the target for an ssh command.
            "dest_host": "lldb-linux.example.com",

            # root_dir specifies the global root directory for
            # this destination.  All destinations on this target
            # will be in a directory that is built from
            # root_dir + base_dir + {dir_id}_dir.
            "root_dir" : "/home/tfiala",

            # The ssh user is specified here.
            "user": "tfiala",

            # The ssh port is specified here.
            "port": 22
        }
    },

    # Describe the lldb-freebsd destination.
    # Very similar to the lldb-linux one.
    {
        "name": "lldb-freebsd"
        "parent": "common_tot",
        "ssh": {
            "dest_host": "lldb-freebsd.example.com",
            # Specify a different destination-specific root dir here.
            "root_dir" : "/mnt/ssd02/fialato",
            "user": "fialato",
            # The ssh port is specified here.
            "port": 2022
        }
    },

    # If a block named "default" exists, and if no configuration
    # is specified on the command line, then the default block
    # will be used.  Use this block to point to the most common
    # transfer destination you would use.
    {
        "name": "default",
        "parent": "lldb-linux"
    }
]
}

Using it

Now that we have a .syncsourcerc file set up, we can do a transfer.
The .syncsourcerc file will be searched for as follows, using the
first one that is found:

* First check the --rc-file RCFILE option.  If this is specified
  and doesn't exist, it will raise an error and quit.

* Check if the current directory has a .syncsourcerc file.  If so,
  use that.

* Use the .syncsourcerc file from the user's home directory.

Run the command:
python /path/to/syncsource.rc -c {configuration-name}

The -c {configuration-name} can be left off, in which case a
configuration with the name 'default' will be used.

After that, the transfer will occur.  With the rsync-over-ssh
transfer agent, one rsync per dir-id will be used.  rsync output
is redirected to the console.

FEEDBACK

Feel free to pass feedback along to Todd Fiala (todd.fiala@gmail.com).