# Hyperopt Example

**URL:** <https://guildai.org/t/hyperopt-example/225>\
**Category:** Examples\
**Created:** [July 10, 2020, 6:34pm UTC](https://guildai.org/t/hyperopt-example/225 "2020-07-10T18:34:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![guildai](https://yyz1.discourse-cdn.com/flex031/user_avatar/guildai.org/guildai/32/103_2.png) [@guildai](https://guildai.org/u/guildai)\
**Post date:** [July 10, 2020, 6:34pm UTC](https://guildai.org/t/hyperopt-example/225/1 "2020-07-10T18:34:22Z")

</div>

## Overview

This [example](https://github.com/guildai/guildai/tree/master/examples/hyperopt) illustrate how to create a custom optimizer using Hyperopt. Follow the patterns outlined below to use other sequential tuning algorithms with your project.

Project files:

| | |
| --- | --- |
| [guild.yml](https://github.com/guildai/guildai/blob/master/examples/hyperopt/guild.yml) | Project Guild file |
| [train.py](https://github.com/guildai/guildai/blob/master/examples/hyperopt/train.py) | Sample training script |
| [tpe.py](https://github.com/guildai/guildai/blob/master/examples/hyperopt/tpe.py) | Optimizer support using Tree of Parzen Estimators with Hyperopt |
| [requirements.txt](https://github.com/guildai/guildai/blob/master/examples/hyperopt/requirements.txt) | List of required libraries |

An [_optimizer_](https://guildai.org/docs/optimization) is a Guild operation that specializes in running multiple trials based on a batch prototype.

In this example, we create a custom optimizer named `tpe`. The optimizer is used to find optimal hyperparameter values using the Tree of Parzen Estimators algorithm from the [Hyperopt](http://hyperopt.github.io/hyperopt/) library.

Here’s the Guild file that defines both the training operation and the optimizer:

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/guild.yml>

> Note Each operation uses the Python module corresponding to its name. If a module uses a different name, specify it using the [`main`](https://guildai.org/reference/guildfile#operation-main) attribute.

Start an optimization batch by running:

```command
guild run train x=[-2:2] --optimize

```

The `train` operation is configured to use the `tpe` optimizer by default (see above).

The `tpe` optimizer is configured to run 10 trials by default. Change this value using `--max-trials`.

You can view operation help for `tpe` to show supported flags. These are set using `-Fo` options with [`guild run`](https://guildai.org/commands/run).

```command
guild run tpe --help-op

```

## Implementation

The `tpe` operation is implemented in [tpe.py](https://github.com/guildai/guildai/blob/master/examples/hyperopt/tpe.py).

Implementation conists of the following steps:

1. Read settings for the batch run and the batch _prototype_.
2. Convert prototype flag values into a Hyperopt [_search space_](https://github.com/hyperopt/hyperopt/wiki/FMin#2-defining-a-search-space)
3. Define a [_function to minimize_](https://github.com/hyperopt/hyperopt/wiki/FMin#1-defining-a-function-to-minimize) that runs a batch trial using hyperparameter values provided by Hyperopt.
4. Use Hyperopt [`fmin`](https://github.com/hyperopt/hyperopt/wiki/FMin) to start a sequential optimization process using the search space and function to minimize.

## Read Batch Settings

Use `batch_util.batch_run()` to get the current batch run. This is used to read the batch settings.

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L24>

## Convert Prototype Flags to Search Space

The batch _proto_ run defines the flag values used for trials. These may contain search specs like `[-2:2]` or `uniform[-2:2]`. This syntax represents a Guild [_search space function_](https://guildai.org/docs/flags#search-space-functions). Decode search space functions to create a Hyperopt search space.

Read the proto flags and uses them to create a search space:

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L69>

Decode each flag function to create a corresponding Hyperopt [search space expression](https://github.com/hyperopt/hyperopt/wiki/FMin#21-parameter-expressions):

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L79>

## Function to Minimize

Hyperopt calls a function with a dictionary of suggested flag values. Use `batch_util.run_trial` to generate a trial for the batch using the suggested values:

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L48>

The result is the objective specified in [`guild run`](https://guildai.org/commands/run) — or `loss` by default.

## Minimize the Objective

Call `fmin` to minimize the objective:

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L32>

### Configure TPE Algorithm

The value for `algo` uses a local function that supports the TPE hyperparameters. The hyperparameters in this case are defined as module global variables. These are imported by Guild and can be customized using `-Fo` options with [`guild run`](https://guildai.org/commands/run).

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L109>

### Perform Summary Actions

This implementation labels the “best” run. Run any number of summary actions in your custom optimzer.

> <https://github.com/guildai/guildai/blob/main/examples/hyperopt/tpe.py#L116>