Migrating from BundleWrap 4.x to 5.x

As per semver, BundleWrap 5.0 breaks compatibility with repositories created for BundleWrap 4.x. This document provides a guide on how to upgrade your repositories to BundleWrap 5.x. Please read the entire document before proceeding.



Changes that require you to adjust your repo


svc_upstart items have been removed

Upstart is a discontinued project, so this item type has been removed.

If you are stuck with upstart for some reason, you will have to copy the last version of svc_upstart.py to your repo as a custom item type.


Hashing now uses sha256

sha1 has been replaced by sha256 in the following places:

  • content_hash of file items. You must update your item definitions.
  • When hashing file contents on the node, sha256 or sha256sum (depending on the OS) is now used. This program must exist on the node.
  • The bw hash ... command will print sha256 hashes now.


repo.nodes_matching() throws NoSuchTarget

When expressions don't match anything, the NoSuchTarget exception will now be thrown. You will have to catch this exception and decide what to do. Previously, this case was silently ignored.


Hooks are called with named arguments

Hooks are called with named arguments now. The names in your function signature must match the ones in our documentation.

For example, you can no longer do this:

def node_apply_start(my_repo, my_node):
    ...

But you must do this instead:

def node_apply_start(repo, node):
    ...


directory and file items auto-depend on corresponding zfs_dataset items

Suppose you have this:

zfs_datasets['tank/foo'] = {
    'mountpoint': '/srv/foo',
}

directories['/srv/foo'] = {
    'needs': {'zfs_dataset:tank/foo'},
}

files['/srv/foo/bar'] = {
    'needs': {'zfs_dataset:tank/foo'},
}

The needs on the directory and file items is no longer necessary, this will be auto-detected now.

(This is not a breaking change, but you should clean this up in order to keep your repo tidy.)


bcrypt

The dependency on passlib has been removed, becase this library is unmaintained.

This affects user items and the as_htpasswd_entry() method of Fault objects. Both will use bcrypt for hashing now, i.e. they look like $2b$12$riWuF3Oh....

The hash_method attribute has been removed from user items.

Current versions of nginx and Apache, and current Linux distributions as well as FreeBSD and OpenBSD should support the $2b$ scheme in their shadow files.

If you do rely on the old behavior, you can still set a user's password_hash manually. There is no replacement regarding as_htpasswd_entry(), you will have to implement your own version of the old crypt algorithms.


node.metadata_get() has been removed

This method was deprecated. Use node.metadata.get() instead.


Canned actions inherit tags

Canned actions like svc_systemd:nginx:restart now inherit the tags from their "parent" item. This mostly affects two scenarios.

Suppose you have a custom item type for an init system and you define a service like this:

svc_fancy_init['nginx'] = {
    'tags': {'causes-downtime'},
}

If your item type supports canned actions like svc_fancy_init:nginx:restart (and if that canned action does not depend on svc_fancy_init:nginx itself), then running the following command will now skip both svc_fancy_init:nginx and the canned action svc_fancy_init:nginx:restart:

$ bw apply mynode -s tag:causes-downtime

Previously, this only skipped svc_fancy_init:nginx. (This does not affect item types from core BundleWrap, because all their canned actions already explicitly depend on the parent item.)

The second scenario is that the following was previously impossible, because the canned actions did not inherit the a tag and thus a dependency loop was created:

svc_systemd = {
    'test.service': {
        'tags': {'a'},
        'needed_by': {'!tag:a'},
    },
}


MetadataUnavailable instead of KeyError

Previously, you could write code like this in metadata reactors:

@metadata_reactor
def my_reactor(metadata):
    for name, config in metadata.get('something', {}).items():
        # key 'ips' may or may not (yet) be available here
        ips = config['ips']

If ips was not available, Python automatically raised a KeyError which would have been caught by BundleWrap and had triggered a reactor re-run at a later point. This is no longer the case.

Instead, we recommend writing code like this:

@metadata_reactor
def my_reactor(metadata):
    for name in metadata.get('something', {}):
        # key 'ips' may or may not (yet) be available here
        ips = metadata.get(f'something/{name}/ips')

Doing it this way will ensure BundleWrap will correctly know which paths were requested, but not yet available and trigger reactor runs accordingly.

(Avoid raising MetadataUnavailable yourself, this should rarely be needed.)


metadata objects don't try to be dicts anymore

This has been deprecated for a long time. If you still have dict-style access like this:

version = metadata['my_program']['version']

Then you must replace it with this, because this is the only public API of these objects now:

version = metadata.get('my_program/version')

Also, metadata.items(), metadata.keys(), and metadata.values() is gone. (If you really must get a flat dict of a node's entire metadata, use metadata.get(tuple()). This is strongly discouraged for performance reasons, though.)

metadata.get('foo') now throws MetadataUnavailable instead of silently returning None. This matches the behavior of other get() operations. For example, metadata.get('foo/var') has already thrown an exception; only "root" access was different.


"Non-reading" metadata reactors raise an exception

When a metadata reactor does not read from metadata but only returns values, BundleWrap now raises an exception. This is a performance-costly anti-pattern and you should use defaults instead.

For example, this:

@metadata_reactor.provides('foo')
def foo(metadata):
    return {'foo': True}

Should be replaced with:

defaults = {
    'foo': True,
}


get_auto_deps() has been replaced by get_auto_attrs()

Custom items could be instructed to auto-discover dependencies. That happened by calling get_auto_deps() on them.

get_auto_deps() has now been replaced by the more generic get_auto_attrs(), which is able to return different kinds of attributes instead of just needs.

To migrate existing code, replace this pattern:

class MyItem(Item):
    def get_auto_deps(self, items):
        # ...
        return [some_item.id, another_item.id]

With this:

class MyItem(Item):
    def get_auto_attrs(self, items):
        # ...
        return {
            'needs': [some_item.id, another_item.id],
        }

Items: display_dicts() → display_on_fix(), keys is a set

Custom items must be updated:

  • The method display_dicts() has been renamed to display_on_fix() for consistency.
  • display_on_fix()'s third argument, keys, is a set now instead of a list.

cdict/sdict moved to properties expected_state/actual_state

The terminology cdict and sdict has been considered confusing. To alleviate this pain, we use expected_state and actual_state now.

Custom items must be updated:

  • Rename cdict to expected_state: This describes the desired state of an item.
  • Rename sdict to actual_state: This describes the actual state of an item on a node.
  • Both these methods need the @property decorator now.



Behavioral changes


Lock identities have changed

If you have locked a node using bw lock and have not overridden your identity using BW_IDENTITY, then the identity $your_username@$your_hostname was used. This has been changed to now include the Git branch name as well (assuming you're using Git): $your_username@$your_hostname:$git_branch_name.

The migration path is to remove the old locks using bw lock remove ... and put new locks on the nodes using bw lock add .... How exactly this needs to be done depends on your use case.


BW_SCP_ARGS defaults to the empty string

This environment variable used to default to the value of BW_SSH_ARGS or, if that was unset, to the empty string. BW_SSH_ARGS is used when calling the ssh binary on your machine, BW_SCP_ARGS for the scp binary. Since some arguments work for both ssh and scp, you might have gotten away with setting just BW_SSH_ARGS (which implicitly also affected scp). This will no longer work.

You must now explicitly set BW_SCP_ARGS.


bw apply and bw verify: -s without matches is an error now

This is an error now:

$ bw verify hw.switch-foobar -s tag:causes-downtime
!!! the following selectors for --skip do not match any items: tag:causes-downtime

The intention is to catch typos. For example, -s tag:causes_downtime (note the underscore instead of a dash) previously went unnoticed and might have unintentionally restarted some services.

This check is done on the entire selection. If you apply a group and no node in that group has items that match the selector, then the error is raised.

There is no direct replacement. If you rely on the old behavior, because you regularly apply nodes where the -s argument doesn't match anything, then you must remove -s in those cases.


Harmonized output of bw items

The output of bw items was harmonized over all subcomannds. Some subcommands that previously generated JSON now default to table output and will need --json to switch back to JSON output.

Tables printed to the CLI with only one column are now formatted as flat list without decorators.

Tables printed to the CLI with the format of BW_TABLE_STYLE=grep were changed to repeat literal columns for every row of an array for all tables (not just some), this might require changes to your scripts if you parse bw command output.

bw item NODE ITEM --state is now called --actual-state to clarify its function.


bw plot --no-depends-reverse has been removed

Instead, reverse dependencies are shown with dashed lines.


bw test always fails if test_with fails

When a test_with command failed, we previously ignored this if its exit code was 126, 127, or 255. This is no longer the case.

These special exit codes usually indicate that a command is not present on a system. The use case was: Run certain test_with commands only on automated build servers, but not on a dev's machine. The reasoning was that some test_with commands might have heavy dependencies that not everybody wants to install on their laptop. This asymmetry – some tests only running on a build server – was confusing, though, because people wondered why their builds failed, even though a local bw test was seemingly successful. Hence this special handling was removed.


bw lock -i $selector: Verifies if $selector matches

To prevent typos and accidents, BundleWrap will now verify if these item selectors match anything. If you don't want this, use --skip-item-verification.


bw lock add: Warning about existing locks might use a pager

bw lock add shows a message like Your lock was added, but the node was already locked by ..., followed by a list of existing locks. This output is piped through a pager now when stdout is a TTY.

This can break your scripts, because they might block now, waiting for the pager to quit. One way to get the old behavior is to override the environment variable PAGER for these calls:

#!/bin/sh

PAGER=cat bw lock add ...



Minor changes

For everything else, please consult the changelog.