# Running jobs show as errors on cluster

**URL:** <https://guildai.org/t/running-jobs-show-as-errors-on-cluster/746>\
**Category:** Troubleshooting\
**Created:** [August 6, 2021, 7:33pm UTC](https://guildai.org/t/running-jobs-show-as-errors-on-cluster/746 "2021-08-06T19:33:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gilded](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gilded](https://guildai.org/u/Gilded)\
**Post date:** [August 6, 2021, 7:33pm UTC](https://guildai.org/t/running-jobs-show-as-errors-on-cluster/746/1 "2021-08-06T19:33:26Z")

</div>

Not sure if this is a bug or a feature request, but here is the issue: I stage a bunch of jobs on computer 1, part of a cluster with a shared file system. When I call `guild runs` on any computer in the cluster, it will properly show all staged files. If I start a queue on computer 1 it will promptly launch the first staged job:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/guild/original/1X/48c319be5e838f51786c86a2f2ff3699552b8cdd.png)

If I then log into computer 2, the running job will show as error:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/guild/original/1X/9464fe7e0aaa6cccbf30dbc317d508cb5ec6718f.png)

Why would it show as error instead of just “running”?

---

<div class="post-metadata">

**Author:** ![garrett](https://yyz1.discourse-cdn.com/flex031/user_avatar/guildai.org/garrett/32/224_2.png) [@garrett](https://guildai.org/u/garrett)\
**Post date:** [January 14, 2022, 4:34pm UTC](https://guildai.org/t/running-jobs-show-as-errors-on-cluster/746/2 "2022-01-14T16:34:14Z")

</div>

Guild determines if a run is “running” by looking at the pid file and checking for a running pid. There are two major problems here:

- A run terminated in a way that leaves its pid file (e.g. power loss, SIGKILL, etc) can later show up as “running” when that pid is recycled
- When checking status on a shared file system, the local process sees a different list of pids

Guild needs to differentiate remote runs from local when checking this status. To get actual bona fide status, Guild needs an interface to the remote system, which it has, but this requires a `guild pull` or `guild sync` op before running `guild runs`.

We’ll take a look at this for either 0.7.5 or 0.8. Thanks for the report and sorry for the super long response time!

---

<div class="post-metadata">

**Author:** ![Gilded](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gilded](https://guildai.org/u/Gilded)\
**Post date:** [January 15, 2022, 6:05pm UTC](https://guildai.org/t/running-jobs-show-as-errors-on-cluster/746/3 "2022-01-15T18:05:50Z")

</div>

No worries, and thanks for the explanation!
