I have multiple datasets that I preprocess using different guild operations. My train operation in turn uses these runs as resources, and I want to be able to only input a selected subset of these preprocess runs to my train operation to be able to experiment with how different data subsets affect t…

Generally you resolve one upstream data prep run for a downstream training run. Guild doesn’t even support resolving multiple runs per resource, unless you explicitly configure it with multiple dependencies. The problem is then how you can select the applicable upstream run to experiment with. The…

Your second method almost solves the problem for me, but I would still like to be able to have the resources be optional. Here’s an example of what I want to do, just to make it clearer. prepare-data-1: main: prepare_data_1 # these aren't versions of the data, but actually completely different …

Ah, I understand better now. Thank you for the clarification. You’ve identified a problem that Guild does not explicitly handle. I think designating a dependency as optional is a possible enhancement we could make. There are some other options we can look at as well. I believe this requirement rel…

Thanks for the reply, the multi-run attribute would be a good solution to my problem! If I understand the idea correctly, it would support supplying a list of runs as the required resource, something like guild run train prepare-data=<id1>,<id2>,<id3>? In my case, the runs would never have name col…

Yes, you have all of that right. The default target path is . (the root run directory) and that would apply to the new multi-run feature. Guild already warns when sources collide so you’d know when that happens. The use of run-specific references (e.g. ${id}) is optional for those who would need it. …

This has been lingering! We have a request for optional for a different use case: [image] Self requiring operation General I am trying to learn a recipe (a sequence of hyperparameters) to train a model. In essence, I would like to have the model requiring itself in a fash…

Will there be an RFC on the optional and multi-feature feature?

How to have optional run resources

General

garrett October 5, 2022, 8:31pm 9

Late late late but here’s the RFC and the feature is committed to main and will be available in 0.8.2 (out in the next week or so)

Topic		Replies	Views
Optional operation/run dependencies RFC	5	377	October 20, 2022
Self requiring operation General	7	968	October 5, 2022
Dependencies Concepts	0	3398	June 12, 2020
How can I define models in guild and run them against different training procedures? General	1	542	March 22, 2022
Change default select rules for operation dependencies RFC	0	297	January 18, 2023

How to have optional run resources

Related topics