Command line tools

Command line interface for migration tools is implemented as a subcommand of butler command. The top-level subcommand name is butler migrate which in turn has few other subcommands. The commands implemented by butler migrate are mostly wrappers for corresponding Alembic commands/scripts, which hide complexity of the Alembic configuration and Butler-specific knowledge.

There are two classes of migrate commands. Commands from the first class work with migration scripts and revision trees, and do not need access to Butler repository. These commands are:

• show-trees

• show-history

• add-tree

• add-revision

Commands from other class work with database and require a Butler repository:

• show-current

• stamp

• upgrade

• downgrade

There are also few commands that were added for specific cases, not generally useful:

• dump-schema

• set-namespace

• rewrite-sqlite-registry

• update-day-obs

Sections below describe individual commands and their options.

Note

Please note that documentation of the default values for some options may refer to a file path at a time of documentation generation, actual default paths will be different when CLI command are executed.

butler migrate

Set of command for managing and applying schema migrations.

Usage

butler migrate [OPTIONS] COMMAND [ARGS]...

add-revision

Create new revision.

TREE_NAME is the revision tree name, usually it is the name of a registry manager type (e.g. ‘datasets’).

MANAGER_CLASS is the name of the manager class (or a namespace in case of special trees), not including module name.

VERSION is the version of manager class, typically in X.Y.Z notation.

Usage

butler migrate add-revision [OPTIONS] TREE_NAME MANAGER_CLASS VERSION

Options

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--one-shot

Use special one-shot history trees.

Arguments

TREE_NAME

Required argument

MANAGER_CLASS

Required argument

VERSION

Required argument

See ‘butler –help’ for more options.

add-tree

Create new revision tree for a specified manager type.

TREE_NAME is the revision tree name, usually it is the name of a registry manager type (e.g. ‘datasets’).

Usage

butler migrate add-tree [OPTIONS] TREE_NAME

Options

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--one-shot

Use special one-shot history trees.

Arguments

TREE_NAME

Required argument

See ‘butler –help’ for more options.

downgrade

Downgrade schema to a specified revision.

REVISION is a target alembic revision, in offline mode it can also specify initial revision using INITIAL:TARGET format.

Usage

butler migrate downgrade [OPTIONS] REPO REVISION

Options

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--one-shot-tree <one_shot_tree>

Use special one-shot history tree instead of regular history.

--sql

Offline mode, dump SQL instead of executing migration on a database.

--namespace <namespace>

Namespace to use when ‘namespace’ key is not present in the stored dimensions configuration

Arguments

REPO

Required argument

REVISION

Required argument

See ‘butler –help’ for more options.

dump-schema

Dump database schema in human-readable format.

TABLE specify multiple optional table names to dump, by default all tables are dumped.

Usage

butler migrate dump-schema [OPTIONS] REPO [TABLE] ...

Arguments

REPO

Required argument

TABLE

Optional argument(s)

See ‘butler –help’ for more options.

rewrite-sqlite-registry

Transfer registry information from one registry to a new registry.

SOURCE is a URI to the Butler repository to be transferred.

On completion a new registry will be written in its place and the old registry moved to a backup file.

Usage

butler migrate rewrite-sqlite-registry [OPTIONS] SOURCE

Arguments

SOURCE

Required argument

See ‘butler –help’ for more options.

set-namespace

Add or update namespace attribute to dimensions.json.

NAMESPACE to add or replace in the stored dimensions configuration. If namespace is not specified then existing namespace is printed.

Usage

butler migrate set-namespace [OPTIONS] REPO [NAMESPACE]

Options

--update

Replace existing namespace if it exists.

Arguments

REPO

Required argument

NAMESPACE

Optional argument

See ‘butler –help’ for more options.

show-current

Display current revisions from either alembic_version or butler_attributes tables.

Usage

butler migrate show-current [OPTIONS] REPO

Options

-v, --verbose

Print detailed information.

--butler

Display butler version numbers for managers from butler_attributes table. By default revisions from alembic_version table are displayed, if that table does not exist the output will be empty.

--namespace <namespace>

Namespace to use when ‘namespace’ key is not present in the stored dimensions configuration

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

Arguments

REPO

Required argument

See ‘butler –help’ for more options.

show-history

Display revision history for a tree.

TREE_NAME is the revision tree name, usually it is the name of a registry manager type (e.g. ‘datasets’).

Usage

butler migrate show-history [OPTIONS] [TREE_NAME]

Options

-v, --verbose

Print detailed information.

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--one-shot

Use special one-shot history trees.

Arguments

TREE_NAME

Optional argument

See ‘butler –help’ for more options.

show-trees

Print a list of known revision trees (manager types).

Usage

butler migrate show-trees [OPTIONS]

Options

-v, --verbose

Print detailed information.

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--one-shot

Use special one-shot history trees.

See ‘butler –help’ for more options.

stamp

Stamp Alembic revision table (alembic_version) with current manager versions from butler_attributes.

MANAGER is a name of the manager for which to stamp the revision, if missing then all managers already defined in butler_attributes are stamped. To stamp initial revision for a manager not in butler_attributes, provide its name explicitly.

Usage

butler migrate stamp [OPTIONS] REPO [MANAGER]

Options

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--purge

Remove existing version table before saving new versions.

--namespace <namespace>

Namespace to use when ‘namespace’ key is not present in the stored dimensions configuration

-n, --dry-run

Do not execute actions, only report.

Arguments

REPO

Required argument

MANAGER

Optional argument

See ‘butler –help’ for more options.

update-day-obs

Update the day_obs values if needed.

INSTRUMENT is the name of the instrument to use.

Usage

butler migrate update-day-obs [OPTIONS] REPO INSTRUMENT

Arguments

REPO

Required argument

INSTRUMENT

Required argument

See ‘butler –help’ for more options.

upgrade

Upgrade schema to a specified revision.

REVISION is a target alembic revision, in offline mode it can also specify initial revision using INITIAL:TARGET format.

Usage

butler migrate upgrade [OPTIONS] REPO REVISION

Options

--mig-path <PATH>

Top-level folder with migration scripts, default: ./migrations

--one-shot-tree <one_shot_tree>

Use special one-shot history tree instead of regular history.

--sql

Offline mode, dump SQL instead of executing migration on a database.

--namespace <namespace>

Namespace to use when ‘namespace’ key is not present in the stored dimensions configuration

--options <TEXT=TEXT>

Options to pass to migration scripts, as a key-value pair.

Arguments

REPO

Required argument

REVISION

Required argument

See ‘butler –help’ for more options.