# Timeout error
**URL:** https://guildai.org/t/timeout-error/196
**Category:** Troubleshooting
**Created:** [June 20, 2020, 12:37pm UTC](https://guildai.org/t/timeout-error/196 "2020-06-20T12:37:23Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@mislav](https://guildai.org/u/mislav)
#### Post date: [June 20, 2020, 12:37pm UTC](https://guildai.org/t/timeout-error/196/1 "2020-06-20T12:37:24Z")
Hi @garrett,
I can’t figure out why sometimes I get timeout error when I run experiments. Here is my running command:
```plaintext
guild run train tb_volatility_lookback=range[30:300:10]
```
it hangs after some time and when I click ctr + c to abort it the below error shows up.
> Traceback (most recent call last):
> File “C:\ProgramData\Anaconda3\lib\runpy.py”, line 193, in \_run\_module\_as\_main
> INFO: [numexpr.utils] NumExpr defaulting to 8 threads.
> “ **main** ”, mod\_spec)
> File “C:\ProgramData\Anaconda3\lib\runpy.py”, line 85, in \_run\_code
> exec(code, run\_globals)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\batch\_main.py”, line 38, in
> main()
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\batch\_main.py”, line 26, in main
> batch\_util.handle\_trials(batch\_run, trials)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\batch\_util.py”, line 54, in handle\_trials
> \_run\_trials(batch\_run, trials)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\batch\_util.py”, line 79, in \_run\_trials
> \_start\_trial\_run(run, stage)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\batch\_util.py”, line 117, in \_start\_trial\_run
> run\_impl.run(restart=run.id, stage=stage)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1940, in run
> main(args)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1017, in main
> \_dispatch\_op(S)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1101, in \_dispatch\_op
> \_dispatch\_op\_cmd(S)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1286, in \_dispatch\_op\_cmd
> \_confirm\_and\_run(S)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1354, in \_confirm\_and\_run
> \_run(S)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1544, in \_run
> \_run\_local(S)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1575, in \_run\_local
> \_run\_op(op, S.args)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\commands\run\_impl.py”, line 1683, in \_run\_op
> extra\_env=extra\_env,
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op.py”, line 160, in run
> exit\_status = \_run(run, op, quiet, stop\_after, extra\_env)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op.py”, line 195, in \_run
> exit\_status = \_op\_wait\_for\_proc(op, proc, run, quiet, stop\_after)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op.py”, line 230, in \_op\_wait\_for\_proc
> return \_op\_watch\_proc(op, proc, run, quiet, stop\_after)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op.py”, line 238, in \_op\_watch\_proc
> return \_proc\_wait(proc, stop\_after)
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op.py”, line 259, in **exit**
> self.\_output.wait\_and\_close()
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op\_util\_legacy.py”, line 254, in wait\_and\_close
> self.close()
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op\_util\_legacy.py”, line 219, in close
> lock = self.\_acquire\_output\_lock()
> File “C:\ProgramData\Anaconda3\lib\site-packages\guild\op\_util\_legacy.py”, line 232, in \_acquire\_output\_lock
> raise RuntimeError(“timeout”)
> RuntimeError: timeout
---
### Author:  [@garrett](https://guildai.org/u/garrett)
#### Post date: [June 20, 2020, 1:30pm UTC](https://guildai.org/t/timeout-error/196/2 "2020-06-20T13:30:57Z")
Is this project generally available where I could try to reproduce this?
This is a deadlock, which is typically hard to track down because it’s hard to recreate. I’d typically ask for a simplified version, but chances are good that as you simplify the project, the issue goes away.
If you’re not able to share the project, we’ll need to come up with a strategy for debugging this.
---
### Author:  [@mislav](https://guildai.org/u/mislav)
#### Post date: [June 20, 2020, 1:38pm UTC](https://guildai.org/t/timeout-error/196/3 "2020-06-20T13:38:06Z")
I should be easy reproducible. Here is my modul [https://github.com/MislavSag/trademl/tree/master/trademl/modeling](https://github.com/MislavSag/trademl/tree/master/trademl/modeling).
In the root the following command should be executed:
guild run --yes --max-trials 150 train labeling\_technique=[trend\_scanning] tb\_volatility\_scaler=[1,1.5,2] tb\_triplebar\_num\_days=[10,20,30,50] tb\_triplebar\_min\_ret=[0.04,0.05,0.06] tb\_volatility\_lookback=range[30:300:10] sample\_weights\_type=[returns,time\_decay]
EDIT: the data is problem! I have data locally. I plan to put it on SQL…
So unfortunately right now it is not possible to reproduce.
---
### Author:  [@garrett](https://guildai.org/u/garrett)
#### Post date: [June 20, 2020, 7:04pm UTC](https://guildai.org/t/timeout-error/196/4 "2020-06-20T19:04:54Z")
I created a [fork](https://github.com/gar1t/trademl) of this project and made some changes so I could run up to the point where the train script needs the h5 data.
If you can recreate this problem with sample data that you can sent me, I’ll see if I can recreate it.
---
### Author:  [@garrett](https://guildai.org/u/garrett)
#### Post date: [June 20, 2020, 7:10pm UTC](https://guildai.org/t/timeout-error/196/5 "2020-06-20T19:10:42Z")
As a work around for this problem, try setting `NO_RUN_OUTPUT=1` when you run the operation.
```command
NO_RUN_OUTPUT=1 guild run train tb_volatility_lookback=range[30:300:10]
```
---
### Author:  [@mislav](https://guildai.org/u/mislav)
#### Post date: [June 21, 2020, 8:50am UTC](https://guildai.org/t/timeout-error/196/6 "2020-06-21T08:50:20Z")
It worked when I used lower number of flags. Maybe there was a problem with one grid search (one flag contained range function), but it works now. I will try to debug it by adding new flags ans see on which one it stops working.
---
### Author:  [@garrett](https://guildai.org/u/garrett)
#### Post date: [June 21, 2020, 2:12pm UTC](https://guildai.org/t/timeout-error/196/7 "2020-06-21T14:12:28Z")
Threading problems are notorious for suddenly appearing and suddenly disappearing. They are highly nondeterministic and therefore hard to track down and fix.
My bet is that you’ll run into this again.
I think the best course would be to produce some sanitized test data (nothing private) that, along with the full set of flags, reproduces the deadlock. With that, hopefully, I can reproduce it on a system. With that I can fix it. Short of that, it’ll be tough.
Otherwise, I suggesting setting `NO_RUN_OUTPUT=1` in your environment to make sure Guild never runs the code in question. With that you can safely run the script without risk of it deadlocking. You can write your output to logs as needed.
---
### Author:  [@mislav](https://guildai.org/u/mislav)
#### Post date: [July 7, 2020, 3:45pm UTC](https://guildai.org/t/timeout-error/196/8 "2020-07-07T15:45:02Z")
@garrett, when I run `NO_RUN_OUTPUT=1` option,
`NO_RUN_OUTPUT=1 guild run --yes --max-trials 32 random_forest_sklearnopt:train num_threads=4 tb_volatility_lookback=[50,200] ts_look_forward_window=[600,1200,2400,4800] sample_weights_type=[returns,time_decay]`
I get the error:
```
NO_RUN_OUTPUT=1 : The term 'NO_RUN_OUTPUT=1' is not recognized as the name of a cmdlet, function, script file, or opera
ble program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ NO_RUN_OUTPUT=1 guild run --yes --max-trials 32 random_forest_sklearn ...
+ ~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (NO_RUN_OUTPUT=1:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
```
---
### Author:  [@garrett](https://guildai.org/u/garrett)
#### Post date: [July 8, 2020, 11:22am UTC](https://guildai.org/t/timeout-error/196/9 "2020-07-08T11:22:27Z")
> [@mislav](#):
>
> `is not recognized as the name of a cmdlet`
This looks like PowerShell. Consult the docs for setting env vars in that shell. [This doc](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-7) from Microsoft seems to apply.
---
### Author:  [@mislav](https://guildai.org/u/mislav)
#### Post date: [July 11, 2020, 2:21pm UTC](https://guildai.org/t/timeout-error/196/10 "2020-07-11T14:21:46Z")
I set it like:
$Env:NO\_RUN\_OUTPUT=1
It works fine on first try. I will see if there will be problems in the future.