# Remotes Reference **URL:** https://guildai.org/t/remotes-reference/172 **Category:** Reference **Created:** [June 13, 2020, 1:53am UTC](https://guildai.org/t/remotes-reference/172 "2020-06-13T01:53:17Z") **Posts on this page:** 1 **Page:** 1
### Author: ![guildai](https://yyz1.discourse-cdn.com/flex031/user_avatar/guildai.org/guildai/32/103_2.png) [@guildai](https://guildai.org/u/guildai) #### Post date: [June 13, 2020, 1:53am UTC](https://guildai.org/t/remotes-reference/172/1 "2020-06-13T01:53:17Z")
## Overview This reference documents remote types, their supported attributes, and provides examples. A remote type is defined using the `type` attribute. The types below are supported. Refer to the applicable section below for details. | **type attribute** | | --- | | [ssh](#ssh) | | [ec2](#ec2) | | [s3](#s3) | | [azure-vm](#azure-vm) | | [azure-blob](#azure-blob) | | [gist](#gist) | ## SSH ### SSH Requirements The following external software is required to use SSH remotes: - SSH service (remote system) - `realpath` command (remote system) - `ssh` command (local system) - `rsync` command (local system) ### SSH Attributes #### description _Remote description (string)_ #### host _Host name to use when connecting to the remote (required string)_ #### port _Port for SSH connections (integer)_ #### user _User for SSH connections (string)_ #### private-key _Local path to the private key for SSH connections (string)_ #### proxy _Proxy to use for the SSH connection (string)_ #### connect-time _Number of seconds to wait for an SSH connection before quitting (integer)_ #### guild-home _Path to Guild home on the remote host (string)_ If this value is not set explicitly, it’s determined from the virtual environment used for the remote or `~/.guild` if a virtual environment is not used. Set this value when Guild home is different from the default location. #### venv-path _Path to a virtual environment on the remote host (string)_ Guild activates the environment for each remote command. Paths are relative to the remote user home directory. Virtual environments are standard Python virtual environments that are created with `virtualenv`, `guild init`, or the Python `venv` module. To strictly control the environment activation, use [`venv-activate`](#ssh-venv-activate) instead. #### guild-env _Alias for [`venv-path`](#ssh-venv-path) (see above)_ #### conda-env _Conda environment name on the remote (string)_ Guild activates the Conda environment for each remote command. If [`guild-env`](#ssh-guild-env) is also specified, the Guild environment is activated rather than the Conda environment. #### venv-activate _Command used to activate an environment on the remote (string)_ Use this to specify the command that Guild uses to activate an environment for remote commands. If this attribute is specified, Guild ignores [`guild-env`](#ssh-guild-env) and [`conda-env`](#ssh-conda-env). #### use-prerelease _Whether to use pre-release versions for required packages (boolean)_ When Guild starts a remote run, it installs the run code along with required Python packages that are specified in the project package definition or in `requirements.txt`. Set this flag to `yes` to instruct Guild to install pre-release versions of required packages. #### init _Shell command to run when the remote is reinitialized (string)_ See [SSH Start and Stop](#ssh-start-and-stop) details information on initializing an SSH remote. ### SSH Status When you run [`guild remote status`](https://guildai.org/commands/remote-status), Guild attempts to connect to the remote host over SSH using the specified connect information. If the connect succeeds, Guild considers the remote to be available. ### SSH Start and Stop SSH remotes cannot be started or stopped. Guild assumes that the specified host is available using the specified connect information. SSH remotes can, however, be initialized using the `--reinit` option to [`guild remote start`](https://guildai.org/commands/remote-start). When `--reinit` is specified, Guild runs the shell command defined in the remote [`init`](#ssh-init) attribute. Consider the following remote configuration: ```yaml remotes: my-remote: type: ssh host: remote-hostname init: | sudo apt update -y sudo apt upgrade -y ``` The command `guild remote start my-remote --reinit` runs the `init` command on the remote host. Use this to ensure that the remote is configured correctly before running Guild commands. ### SSH Remote Commands The commands below can be used with SSH remotes. | | | | --- | --- | | [`check`](https://guildai.org/commands/check) | _Check Guild on the remote_ | | [`run`](https://guildai.org/commands/run) | _Run an operation on a remote_ | | [`stop`](https://guildai.org/commands/runs-stop) | _Stop runs in progress on a remote_ | | [`watch`](https://guildai.org/commands/watch) | _Connect to a remote run in progress and watch its output_ | | [`runs`](https://guildai.org/commands/runs-list) | _List runs on a remote_ | | [`runs info`](https://guildai.org/commands/runs-info) | _Show information about a remote run_ | | [`ls`](https://guildai.org/commands/ls) | _List remote run files_ | | [`diff`](https://guildai.org/commands/diff) | _Diff remote runs_ | | [`cat`](https://guildai.org/commands/cat) | _Show remote run file or output_ | | [`label`](https://guildai.org/commands/label) | _Apply a label to one or more remote runs_ | | [`runs delete`](https://guildai.org/commands/runs-delete) | _Delete remote runs_ | | [`runs restore`](https://guildai.org/commands/runs-restore) | _Restore deleted remote runs on a remote_ | | [`runs purge`](https://guildai.org/commands/runs-purge) | _Purge deleted remote runs on a remote_ | | [`pull`](https://guildai.org/commands/pull) | _Copy remote runs to the local environment_ | | [`push`](https://guildai.org/commands/push) | _Copy local runs to the remote_ | ### SSH Security Security is managed by the SSH protocol between the local system (client) and the remote (server). Connection settings are defined as attributes (see above) and by the SSH environment. Refer to the SSH documentation for your system for additional security details. ### SSH Examples Refer to [_Remotes Cheatsheet_](https://guildai.org/cheatsheets/remotes#ssh) for SSH configuration examples. ## EC2 EC2 provides the same features as SSH remotes. In addition, EC2 remotes can be started and stopped on EC2 by configuring EC2 instance attributes. Refer to [Attributes](#ec2-attributes) below for details. ### EC2 Requirements The following external software is required to use EC2 remotes: - [AWS Command Line Interface](https://aws.amazon.com/cli/) ### EC2 Attributes Any [SSH Attribute](#ssh-attributes) may be used to configure an EC2 remote. Refer to that section for details. In addition to supporting SSH attributes, EC2 remotes support various EC2-specific settings. These are used when running [`guild remote start`](https://guildai.org/commands/remote-start) to create new EC2 resources to support remote operations. #### ami _AMI used to create the EC2 instance (required string)_ #### instance-type _Type of EC2 instance to create (required string)_ #### region _AWS region to create the EC2 instance in (string)_ If this value isn’t specified, Guild uses the value defined by the `AWS_DEFAULT_REGION` environment variable. #### root-device-size _Size of the root volume created for the server (integer)_ If this value is omitted, the default volume size for the AMI is used. #### public-key _The public key used when starting the EC2 instance (string or file path)_ This value may be a path, relative to the user configuration directory (i.e. `~/.guild`) or the base 64 encoded public key. The public key is installed on the new instance to provide SSH access. This public key must correspond to the private key used to access the instance (see [`private-key`](#ec2-private-key) below). #### private-key _The private key used to connect to the VM instance (string or file path)_ This value may be a path, relative to the user configuration directory (i.e. `~/.guild`) or the base 64 encoded private key. #### init-timeout _The timeout to wait for a connection to the instance to become available (integer or string)_ If this value is an integer, it specifies the timeout in minutes. If the timeout is a string, it myst be a number followed by the letter ‘s’ or ‘m’, to indicate seconds or minutes respectively. If a valud is not specified, 5 minutes is assumed. ### EC2 Status On [`guild remote status`](https://guildai.org/commands/remote-status), Guild attempts to connect to the remote host over SSH using the specified connect information. If the connect succeeds, Guild considers the remote to be available. ### EC2 Start and Stop On [`guild remote start`](https://guildai.org/commands/remote-start), Guild uses [Terraform](https://terraform.io) to start an EC2 instance using the EC2 remote settings. On [`guild remote stop`](https://guildai.org/commands/remote-stop), Guild similarly terminates the EC2 instance using Terraform. ### EC2 Remote Commands The commands below can be used with EC2 remotes. | | | | --- | --- | | [`check`](https://guildai.org/commands/check) | _Check Guild on the remote_ | | [`run`](https://guildai.org/commands/run) | _Run an operation on a remote_ | | [`stop`](https://guildai.org/commands/runs-stop) | _Stop runs in progress on a remote_ | | [`watch`](https://guildai.org/commands/watch) | _Connect to a remote run in progress and watch its output_ | | [`runs`](https://guildai.org/commands/runs-list) | _List runs on a remote_ | | [`runs info`](https://guildai.org/commands/runs-info) | _Show information about a remote run_ | | [`ls`](https://guildai.org/commands/ls) | _List remote run files_ | | [`diff`](https://guildai.org/commands/diff) | _Diff remote runs_ | | [`cat`](https://guildai.org/commands/cat) | _Show remote run file or output_ | | [`label`](https://guildai.org/commands/label) | _Apply a label to one or more remote runs_ | | [`runs delete`](https://guildai.org/commands/runs-delete) | _Delete remote runs_ | | [`runs restore`](https://guildai.org/commands/runs-restore) | _Restore deleted remote runs on a remote_ | | [`runs purge`](https://guildai.org/commands/runs-purge) | _Purge deleted remote runs on a remote_ | | [`pull`](https://guildai.org/commands/pull) | _Copy remote runs to the local environment_ | | [`push`](https://guildai.org/commands/push) | _Copy local runs to the remote_ | ### EC2 Security The following environment variables must be defined when starting or stopping an EC2 remote. | | | | --- | --- | | `AWS_ACCESS_KEY_ID` | The AWS access key ID that has the requisite permissions for starting or terminating the applicable EC2 instance. | | `AWS_SECRET_ACCESS_KEY` | The AWS secret access key associated with the access key ID. | The following environment variables may be optionally defined: | | | | --- | --- | | `AWS_DEFAULT_REGION` | The AWS region the remote instance is created in. This value may alternatively be defined in the EC2 remote configuration. | Once an EC2 remote is created, security is managed by the SSH protocol between the local system (client) and the remote (server). ### EC2 Examples Refer to [_Remotes Cheatsheet_](https://guildai.org/cheatsheets/remotes#ec2) for EC2 configuration examples. ## S3 Use S3 remotes to store runs remotely. S3 buckets support the full set of remote management commands. See [Remote Commands](#s3-remote-commands-23) below for details. ### S3 Requirements The following external software is required to use SSH remotes: - [AWS Command Line Interface](https://aws.amazon.com/cli/) ### S3 Attributes #### description _Remote description (string)_ #### bucket _S3 bucket to store runs in (required string)_ #### root _Path in the S3 bucket to store runs in (string)_ #### region _AWS region where the bucket was created (string)_ The region may alternatively be specified using the `AWS_DEFAULT_REGION` environment variable when running a remote command for the S3 remote. ### S3 Status When you run [`guild remote status`](https://guildai.org/commands/remote-status), Guild checks that the bucket exists and that Guild can read from it. This check does not perform a write test. ### S3 Start and Stop S3 remotes can be started and stopped. When started, Guild creates the remote bucket as needed. When stopped, Guild deletes the remote bucket if it exists. > Important Stopping an S3 remote will delete all contents of the remote bucket, including Guild runs any objects that were copied to the bucket independently of Guild. This operation cannot be undone. ### S3 Remote Commands The following commands can be used with S3 remotes. | | | | --- | --- | | [`runs`](https://guildai.org/commands/runs-list) | _List runs on a remote_ | | [`runs info`](https://guildai.org/commands/runs-info) | _Show information about a remote run_ | | [`runs delete`](https://guildai.org/commands/runs-delete) | _Delete remote runs_ | | [`runs restore`](https://guildai.org/commands/runs-restore) | _Restore deleted remote runs on a remote_ | | [`runs purge`](https://guildai.org/commands/runs-purge) | _Purge deleted remote runs on a remote_ | | [`pull`](https://guildai.org/commands/pull) | _Copy remote runs to the local environment_ | | [`push`](https://guildai.org/commands/push) | _Copy local runs to the remote_ | ### S3 Security The following environment variables must be defined when running any command on an S3 remote. | | | | --- | --- | | `AWS_ACCESS_KEY_ID` | The AWS access key ID that has the requisite permissions on the remote S3 bucket. | | `AWS_SECRET_ACCESS_KEY` | The AWS secret access key associated with the access key ID. | The following environment variables may be optionally defined: | | | | --- | --- | | `AWS_DEFAULT_REGION` | The AWS region associated with the bucket. This value may alternatively be defined in the S3 remote configuration. | ### S3 Examples Refer to [_Remotes Cheatsheet_](https://guildai.org/cheatsheets/remotes#s3) for S3 configuration examples. ## Azure VM The Azure VM remote type provides the same features as SSH remotes. In addition, Azure VM remotes can be started and stopped on Azure by configuring Azure VM instance attributes. Refer to [Attributes](#azure-vm--attributes) below for details. ### Azure VM Requirements The following external software is required to use Azure VM remotes: - [Azure Command Line Interface](https://docs.microsoft.com/en-us/cli/azure/) ### Azure VM Attributes Any [SSH Attribute](#ssh-attributes) may be used to configure an Azure VM remote. Refer to that section for details. In addition to supporting SSH attributes, EC2 remotes support various EC2-specific settings. These are used when running [`guild remote start`](https://guildai.org/commands/remote-start) to create new EC2 resources to support remote operations. #### image _Azure image ID used to create the VM instance (required string)_ #### instance-type _Type of Azure VM instance to create (required string)_ #### root-device-size _Size of the root volume created for the server (integer)_ If this value is omitted, the default volume size for the image is used. #### disk-type _The type of storage to use for the managed disk (string)_ If this value is omitted, `Premium_LRS` is used. For a list of supported values, refer to the Terraform [`storage_account_type`](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/managed_disk#storage_account_type) attribute. #### public-key _The public key used when starting the VM instance (string or file path)_ This value may be a path, relative to the user configuration directory (i.e. `~/.guild`) or the base 64 encoded public key. The public key is installed on the new instance to provide SSH access. This public key must correspond to the private key used to access the instance (see [`private-key`](#azure-vm-private-key) below). #### private-key _The private key used to connect to the VM instance (string or file path)_ This value may be a path, relative to the user configuration directory (i.e. `~/.guild`) or the base 64 encoded private key. #### init-timeout _The timeout to wait for a connection to the instance to become available (integer or string)_ If this value is an integer, it specifies the timeout in minutes. If the timeout is a string, it myst be a number followed by the letter ‘s’ or ‘m’, to indicate seconds or minutes respectively. If a valud is not specified, 5 minutes is assumed. ### Azure VM Status On [`guild remote status`](https://guildai.org/commands/remote-status), Guild attempts to connect to the remote host over SSH using the specified connect information. If the connect succeeds, Guild considers the remote to be available. ### Azure VM Start and Stop On [`guild remote start`](https://guildai.org/commands/remote-start), Guild uses [Terraform](https://terraform.io) to start an Azure VM instance using the Azure VM remote settings. On [`guild remote stop`](https://guildai.org/commands/remote-stop), Guild similarly terminates the Azure VM instance using Terraform. ### Azure VM Remote Commands The commands below can be used with Azure VM remotes. | | | | --- | --- | | [`check`](https://guildai.org/commands/check) | _Check Guild on the remote_ | | [`run`](https://guildai.org/commands/run) | _Run an operation on a remote_ | | [`stop`](https://guildai.org/commands/runs-stop) | _Stop runs in progress on a remote_ | | [`watch`](https://guildai.org/commands/watch) | _Connect to a remote run in progress and watch its output_ | | [`runs`](https://guildai.org/commands/runs-list) | _List runs on a remote_ | | [`runs info`](https://guildai.org/commands/runs-info) | _Show information about a remote run_ | | [`ls`](https://guildai.org/commands/ls) | _List remote run files_ | | [`diff`](https://guildai.org/commands/diff) | _Diff remote runs_ | | [`cat`](https://guildai.org/commands/cat) | _Show remote run file or output_ | | [`label`](https://guildai.org/commands/label) | _Apply a label to one or more remote runs_ | | [`runs delete`](https://guildai.org/commands/runs-delete) | _Delete remote runs_ | | [`runs restore`](https://guildai.org/commands/runs-restore) | _Restore deleted remote runs on a remote_ | | [`runs purge`](https://guildai.org/commands/runs-purge) | _Purge deleted remote runs on a remote_ | | [`pull`](https://guildai.org/commands/pull) | _Copy remote runs to the local environment_ | | [`push`](https://guildai.org/commands/push) | _Copy local runs to the remote_ | ### Azure VM Security The following environment variables must be defined when starting or stopping an Azure VM remote. | | | | --- | --- | | `AWS_ACCESS_KEY_ID` | The AWS access key ID that has the requisite permissions for starting or terminating the applicable Azure VM instance. | | `AWS_SECRET_ACCESS_KEY` | The AWS secret access key associated with the access key ID. | The following environment variables may be optionally defined: | | | | --- | --- | | `AWS_DEFAULT_REGION` | The AWS region the remote instance is created in. This value may alternatively be defined in the Azure VM remote configuration. | Once an Azure VM remote is created, security is managed by the SSH protocol between the local system (client) and the remote (server). ### Azure VM Examples Refer to [_Remotes Cheatsheet_](https://guildai.org/cheatsheets/remotes#azure-vm) for Azure VM configuration examples. ## Azure Blob Use Azure Blob remotes to store runs remotely in [Microsoft Azure blob storage](https://azure.microsoft.com/en-us/services/storage/blobs/). The Azure Blob remote supports a subset of remote management commands. See [Remote Commands](#azure-blob-remote-commands) below for details. ### Azure Blob Requirements The following external software is required to use Azure Blob remotes: - [AzCopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) ### Azure Blob Attributes #### description _Remote description (string)_ #### container _Azure blob container URI to store runs in (required string)_ Refer to [_Resource URI Syntax_](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#resource-uri-syntax) for Azure resources for information on the URI format. #### root _Path in the Azure container to store runs in (string)_ ### Azure Blob Status When you run [`guild remote status`](https://guildai.org/commands/remote-status), Guild checks that the container exists and that Guild can read from it. This check does not perform a write test. ### Azure Blob Start and Stop Azure Blob remotes do not currently support [`guild remote start`](https://guildai.org/commands/remote-start) and [`guild remote stop`](https://guildai.org/commands/remote-stop). You must create and delete the applicable containers outside of Guild. Refer to [_Quickstart: Upload, download, and list blobs with the Azure portal_](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal) for more information. ### Azure Blob Remote Commands The Azure Blob remote does not support all remote management commands. In particular, Azure Blob remotes do not support non-permanent deletes. Therefore, [`runs restore`](https://guildai.org/commands/runs-restore) and [`runs purge`](https://guildai.org/commands/runs-purge) are not supported. Additionally, [`runs delete`](https://guildai.org/commands/runs-delete) requires the `--permanent` option. The following commands can be used with Azure Blob remotes: | | | | --- | --- | | [`runs`](https://guildai.org/commands/runs-list) | _List runs on a remote_ | | [`runs info`](https://guildai.org/commands/runs-info) | _Show information about a remote run_ | | [`runs delete`](https://guildai.org/commands/runs-delete) | _Delete remote runs_ | | [`pull`](https://guildai.org/commands/pull) | _Copy remote runs to the local environment_ | | [`push`](https://guildai.org/commands/push) | _Copy local runs to the remote_ | The following commads are not currently supported for Azure Blob remotes: | | | | --- | --- | | [`runs restore`](https://guildai.org/commands/runs-restore) | _Restore deleted remote runs on a remote — not supported_ | | [`runs purge`](https://guildai.org/commands/runs-purge) | _Purge deleted remote runs on a remote — not supported_ | ### Azure Blob Security Security is handled via [AzCopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) authorization using Azure Active Directory. You must log in using `azcopy login` before running Guild commands that access the applicable remote container. For more information, see [_Authorize access to blobs with AzCopy and Azure Active Directory (Azure AD)_](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory). Guild does not support Azure SAS access tokens at this time. ### Azure Blob Examples Refer to [_Remotes Cheatsheet_](https://guildai.org/cheatsheets/remotes#azure-blob) for Azure Blob configuration examples. ## Gist The Gist remote type is used to store Guild runs in a [GitHub gist](https://docs.github.com/en/free-pro-team@latest/github/writing-on-github/creating-gists). This is most convenient with Guild’s inline remote support. Provided you have the required Guild create gists with public access rights. To make a gist private, you must [change the gist visibility](https://github.blog/2014-05-09-change-the-visibility-of-your-gists/). ### Gist Requirements The following external software is required to use Gist remotes: - [git](https://git-scm.com/) ### Gist Attributes Attributes #### description _Remote description (string)_ #### user _Gist owner GitHub user name (required string)_ #### gist-name _Name to use for the gist (required string)_ This value must be unique for `user`. Guild uses this attribute to generate a README file in the gist to identify it. ### Gist Status When you run [`guild remote status`](https://guildai.org/commands/remote-status), Guild checks that gist exists and that Guild can read from it. This check does not perform a write test. ### Gist Start and Stop Guild implicitly creates the applicable gist in GitHub whenever runs are pushed to it. You may explicitly create a gist using using [`guild remote start`](https://guildai.org/commands/remote-start). Delete a gist using [`guild remote stop`](https://guildai.org/commands/remote-stop). Note that when a gist is deleted, any associated runs cannot be recovered. ### Gist Remote Commands The Gist remote does not support all remote management commands. In particular, Gist remotes do not support non-permanent deletes. Therefore, [`runs restore`](https://guildai.org/commands/runs-restore) and [`runs purge`](https://guildai.org/commands/runs-purge) are not supported. Additionally, [`runs delete`](https://guildai.org/commands/runs-delete) requires the `--permanent` option. The following commands can be used with Gist remotes: | | | | --- | --- | | [`runs`](https://guildai.org/commands/runs-list) | _List runs on a remote_ | | [`runs info`](https://guildai.org/commands/runs-info) | _Show information about a remote run_ | | [`runs delete`](https://guildai.org/commands/runs-delete) | _Delete remote runs_ | | [`pull`](https://guildai.org/commands/pull) | _Copy remote runs to the local environment_ | | [`push`](https://guildai.org/commands/push) | _Copy local runs to the remote_ | The following commads are not currently supported for Gist remotes: | | | | --- | --- | | [`runs restore`](https://guildai.org/commands/runs-restore) | _Restore deleted remote runs on a remote — not supported_ | | [`runs purge`](https://guildai.org/commands/runs-purge) | _Purge deleted remote runs on a remote — not supported_ | ### Gist Security Guild creates publicly visible gists. Therefore anyone can read published runs without authorization. You may [make a gist private](https://github.blog/2014-05-09-change-the-visibility-of-your-gists/) after it’s created to limit access. To create or write to a gist, you must provide a gist-authorized [personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token) using the environment variable `GIST_ACCESS_TOKEN`. By default, Guild uses the `https` URL type to access gists. If you prefer to to use `ssh` URL types, define the environment variable `GIST_URLTYPE` to be `ssh`. ### Gist Blob Examples Refer to [_Remotes Cheatsheet_](https://guildai.org/cheatsheets/remotes#gist) for Gist configuration examples. ### Gist Inline Specs A Gist remote may be specified using an inline specification. The format is: ``` gist:USER/GIST_NAME ``` For example, to push runs to a gist named `shared-runs` for the GitHub user `jack`, run: ```command guild push gist:jack/shared-runs ``` To write to a gist, you must have write access provided by a personal access token specified by the environment variable `GIST_ACCESS_TOKEN`. You may omit the user from a gist spec by defining the `GIST_USER` environment variable.