Hi @garrett , I can’t figure out why sometimes I get timeout error when I run experiments. Here is my running command: guild run train tb_volatility_lookback=range[30:300:10] it hangs after some time and when I click ctr + c to abort it the below error shows up. Traceback (most recent call last)…

Is this project generally available where I could try to reproduce this? This is a deadlock, which is typically hard to track down because it’s hard to recreate. I’d typically ask for a simplified version, but chances are good that as you simplify the project, the issue goes away. If you’re not ab…

I should be easy reproducible. Here is my modul https://github.com/MislavSag/trademl/tree/master/trademl/modeling . In the root the following command should be executed: guild run --yes --max-trials 150 train labeling_technique=[trend_scanning] tb_volatility_scaler=[1,1.5,2] tb_triplebar_num_days=[…

I created a fork of this project and made some changes so I could run up to the point where the train script needs the h5 data. If you can recreate this problem with sample data that you can sent me, I’ll see if I can recreate it.

As a work around for this problem, try setting NO_RUN_OUTPUT=1 when you run the operation. NO_RUN_OUTPUT=1 guild run train tb_volatility_lookback=range[30:300:10]

It worked when I used lower number of flags. Maybe there was a problem with one grid search (one flag contained range function), but it works now. I will try to debug it by adding new flags ans see on which one it stops working.

Threading problems are notorious for suddenly appearing and suddenly disappearing. They are highly nondeterministic and therefore hard to track down and fix. My bet is that you’ll run into this again. I think the best course would be to produce some sanitized test data (nothing private) that, alon…

@garrett , when I run NO_RUN_OUTPUT=1 option, NO_RUN_OUTPUT=1 guild run --yes --max-trials 32 random_forest_sklearnopt:train num_threads=4 tb_volatility_lookback=[50,200] ts_look_forward_window=[600,1200,2400,4800] sample_weights_type=[returns,time_decay] I get the error: NO_RUN_OUTPUT=1 : The ter…

[image] mislav: is not recognized as the name of a cmdlet This looks like PowerShell. Consult the docs for setting env vars in that shell. This doc from Microsoft seems to apply.

Timeout error

Troubleshooting

mislav July 11, 2020, 2:21pm 10

I set it like:
$Env:NO_RUN_OUTPUT=1

It works fine on first try. I will see if there will be problems in the future.

NameError: guild doesnt recognize defined variable?

Topic		Replies	Views
Guild run hangs / very slow Troubleshooting	2	656	October 8, 2020
NameError: guild doesnt recognize defined variable? Troubleshooting	9	1775	July 23, 2020
Get Started: Use Guild in a Project Get Started	0	5931	June 27, 2020
Runs Concepts	0	7523	June 7, 2020
Strange out-of-memory behavior on Guild with XGBoost Troubleshooting	7	628	November 10, 2022

Timeout error

Related topics