How do I use variadic args?

Troubleshooting
1

I’m using a Python argument parser that would like to take multiple directories as input, so I have:

parser = argparse.ArgumentParser()
parser.add_argument(
    "--model_dirs", action="store", type=str, help="Path to trained models.", nargs="+", required=True
)

It appears this is supported because I see some usage of it in an example: https://github.com/guildai/guildai/blob/2cc91df428d53c403962f300d3173f319dae782d/examples/detectron2/demo.py#L47

How do I pass multiple values in the CLI so that Guild will separate them?

guild run ... model_dirs="foo bar" -> args.model_dirs = ["foo bar"]
guild run ... model_dirs=foo bar -> errors
guild run ... "model_dirs=foo bar" -> args.model_dirs = ["foo bar"]

2

Hmm - that example is misleading. That file is copied from the upstream sources but isn’t used by Guild in the way you’re expecting there.

Guild doesn’t support multiple flag values as a single assignment. This has come up a few times before — I believe there’s a GitHub issue for that. I’ll revive that issue and copy you there so you have a reference to it.

This is tricky for Guild (I think) as Guild doesn’t support lists for flag values, which is what you’re talking about here.

In the meantime, the official approach for this is to use a single string value and parse the value as you see fit. An easy way to enable support for a list is to use Python’s shlex module to parse a string.

With this example you need to remote nargs='+' in your arg config (this breaks you argparse interface though, not so great):

import shlex

model_dirs = shlex.split(args.model_dirs)

To support nargs='+' with Guild:

if os.getenv("GUILD_OP") and args.model_dirs:
    args.model_dirs = shlex.split(args.model_dirs[0])

I think the later is a better pattern as it preserves your intended interface. The use of Guild is a change, but as Guild formalizes flags as an interface (rather than args) I think this is okay. Of course what’s not okay is the modification to code for Guild’s sake, which we want to avoid.

Implementation Notes

Flag attr is-list

I can imagine a flag attr is-list (boolean) that enables this behavior automatically. Guild would detect this when importing the flag when nargs is a list-enabling value. The question I have is whether this string-parsing via shlex is the right interface.

The list notation (e.g. model_dirs=[a,b,c]) is used for generating trials. We can’t use that to get a list value for a single run. I’ve wondered what interface we should provide. Introducing another list specification is tricky. We’re already annoying zsh users with the use of [...] as zsh treats square brackets as tokens. Other tokens to avoid: (...), {...}, #...# — the list goes on :slight_smile:

Then there’s the issue of lists of lists for multiple runs. E.g. model_dirs=[[a,b,c],[d,e,f]].

Using the shell parsing logic via shlex.split is a way to handle this, as long as Guild knows the flag is a list. I think this is reasonable though — the flag attr type overrides the default parsing rules. So is-list tells Guild to treat values as lists, parsing them with Python’s shlex.split(VAL, posix=True) algorithm.

Flag attr parser

This could be spelled as a “command line arg parser” rather than “is a list”. This is independent of type, which gives a hint as to how the value might be parsed (e.g. list of int, etc.)

train:
  flags:
    model_dirs:
      parser: shlex

The value for parser could be a reference to a Python function (e.g. utils.parse_model_dirs). The function signature would be (arg_val) or (arg_val, flag_def) (i.e. arity of 1 or 2).

If a string, the reference could be a reference to guild.flag_util.parse_<str>. So e.g. shlex would invoke guild.flag_util.parse_shlex.

The advantage of this approach over is-list is obviously flexibility — users can concoct their own flag parsing routines as needed (e.g. sets, maps, JSON, etc.) The spelling parser: shlex also provides the important detail that the strings must be formatted as shlex-parseable strings, which is otherwise implicit.

The counterpoint is that this is complicating things to accommodate edge cases. And while this scheme works nicely with Python globals, it does not work seamlessly with argparse or other command line arg interfaces. A simple “list plus type” scheme does. E.g. argparse doesn’t support maps or other non-list structures.

The only case that occurs to me where parser is not a stretch to specify a different list interface. E.g. parser: csv would indicate that values are parsed using comma delimiters.

3

:+1:

I ended up passing a directory, and using a glob pattern to filter things underneath.

My use case here is that I have trained multiple models with Guild, and want to obtain some summary statistics over all the models trained: e.g. averaging accuracies, plotting all the confusion matrices, plotting ROC curves for all models together.

Right now my approach is to mark the desired runs and export them to a directory. My script then takes the directory, and processes all the run directories.

1 Like
4

Hah, you’ve identified another planned feature in your use case!

Guild (0.7) lacks a coherent strategy to support summary operations across multiple runs. I’d like to see an easy for for users to perform an operation across a set of runs.

Implementation Notes

This strikes me as a map-reduce problem.

My quick thoughts on this:

  • Should be possible to control the set of runs being summarized from the command line — i.e. this logic should not be hard-coded in a script.
  • The summary operation would receive some structure to iterate over, or a callback with state.
  • The summary results should be available as run output, scalars, etc. — i.e. the summary run is like any other run.
  • The interface should not require a Guild library import, though Guild should provide a simple API to support this. WSGI is a good example of a framework agnostic API in Python.

Util support for main module

Following the batch util pattern (used for batches/optimizers):

summary-op:
  main: summary_main

# summary_main.py

from guild import summary_util

def summarize(run, state):
    state.setdefault("run-ids", []).append(run.id)

if __name__ == "__main__":
    result = summary_util.summarize(summarize, {})
    print(result["run-ids"])

Alternatively:

# summary_main.py (alt interface)

from guild import summary_util

if __name__ == "__main__":
    run_ids = [run.id for run in summary_util.iter_runs()]
    print(run_ids)

Guild independent Python interface (novel)

The intent of this interface is to provide an interface that’s independent of Guild. The approach, naming conventions, etc. are entirely speculative at this point.

summary-op:
    summary-callbacks: summary

# summary.py

def summary_init(parent_run):
    return {}

def summary_run(run, state):
    state.setdefault("run-ids", []).append(run["id"])

def summary_finish(state):
    print(state["run-ids"])

We need three functions: init state, handle run, and finish. The example above specifies a Python module. Guild should assume three functions by default when a single item is specified (e.g. summary_init/1, summary_run/2, and summary_finish/1 above). The value could alternatively be a three-tuple where each function is specified explicitly.

Run arguments to these functions should be Python dict-like object used to access run-to-be-summarized attributes. This should not be a Guild-specific interface (though Guild might provide a lazy loading optimization). This follows the WSGI environ pattern where request state is provided by a framework neutral data structure.

This approach introduces a new pattern: the callback interface. I’d expect this to be applied to batch/optimizers as well.

2 Likes