Pipenv compatibility in 0.9.0

Troubleshooting
1

I’ve had some unexpected behavior using pipenv in 0.9.0 versus 0.8.2 - using guild check yields the following:

  • in 0.9.0
➤ pipenv run guild check
guild_version:             0.9.0
guild_install_location:    /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/lib/python3.11/site-packages/guild
guild_home:                /home/neelav/.guild
guild_resource_cache:      /home/neelav/.guild/cache/resources
installed_plugins:         config_flags, cpu, dask, disk, dvc, exec_script, gpu, ipynb, keras, memory, perf, python_script, quarto_document, queue, r_script, resource_flags, skopt
python_version:            3.11.3 (main, Apr  5 2023, 15:52:25) [GCC 12.2.1 20230201]
python_exe:                /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/bin/python
platform:                  Linux 6.3.1-arch2-1 x86_64
psutil_version:            5.9.5
tensorboard_version:       2.13.0
cuda_version:              12.1
nvidia_smi_version:        530.41.03
latest_guild_version:      0.9.0
  • in 0.8.2:
➤ pipenv run guild check
guild_version:             0.8.2
guild_install_location:    /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/lib/python3.11/site-packages/guild
guild_home:                /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/.guild
guild_resource_cache:      /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/.guild/cache/resources
installed_plugins:         config_flags, cpu, dask, disk, dvc, exec_script, gpu, ipynb, keras, memory, perf, python_script, quarto_document, queue, r_script, skopt
python_version:            3.11.3 (main, Apr  5 2023, 15:52:25) [GCC 12.2.1 20230201]
python_exe:                /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/bin/python
platform:                  Linux 6.3.1-arch2-1 x86_64
psutil_version:            5.9.5
tensorboard_version:       2.13.0
cuda_version:              12.1
nvidia_smi_version:        530.41.03
latest_guild_version:      0.9.0
A newer version of Guild AI is available. Run 'pip install guildai --upgrade' to install it.

so in 0.9.0 the guild_home variable doesn’t seem to point to the virtual environment like it does in 0.8.2, instead pointing at my user’s .guild directory. Moreover, I seem to run into issues with guild moving my sourcecode files into the run directory. I have a small example of this with the following guild file:

train:
    exec: python .guild/sourcecode/train.py
    output-scalars: False 
    sourcecode:
        - exclude: '*'
        - '*.py'
    requires:
        -
            file: guild.yml
            target-type: copy

and a Hello World script.

def main():
    print("Hello World")

if __name__ == '__main__':
    main()

The result:

➤ pipenv shell
Launching subshell in virtual environment...
Welcome to fish, the friendly interactive shell
Type help for instructions on how to use fish
➤  source /home/neelav/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/bin/activate.fish
➤ guild run train
You are about to run train
Continue? (Y/n) 
Resolving file:guild.yml
python: can't open file '/home/neelav/.guild/runs/d6d109936802465493d5ea91a3439f98/.guild/sourcecode/train.py': [Errno 2] No such file or directory

and ls on the run directory gives:

➤ ls -a ~/.guild/runs/d6d109936802465493d5ea91a3439f98/.guild/
./  ../  attrs/  manifest  opref  output  output.index

In contrast, using version 0.8.2 with this configuration:

➤ guild run train
You are about to run train
Continue? (Y/n) 
Resolving file:guild.yml
Hello World
➤ guild runs
[1:4c264831]  train  2023-05-15 17:34:11  completed  
➤ guild ls 1
~/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/.guild/runs/4c264831b090451d86592dae5d359f3a:
  guild.yml
➤ ls -a ~/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/.guild/runs/4c264831b090451d86592dae5d359f3a
./  ../  .guild/  guild.yml
➤ ls -a ~/.local/share/virtualenvs/guild_0.9.0_demo-UrtH4SR2/.guild/runs/4c264831b090451d86592dae5d359f3a/.guild/
./  ../  attrs/  manifest  opref  output  output.index  sourcecode/

So something seems to have changed regarding how guild interacts with pipenv between 0.8.2 and 0.9.0. Are there changes to the guild file that I need to make in 0.9.0 to get that functionality from 0.8.2?

2

Hello! My apologies for the late reply here…

You’re running into two changes in 0.9:

  1. In 0.9, Guild treats .guild directories in the current directory as ‘Guild home’ by default. To use an explicit location, set GUILD_HOME. Alternatively, modify ~/.guild/config.yml to include:
# ~/.guild/config.yml

legacy:
  guild-home: pre-0.9

If this legacy behavior should be applied to specific projects, create guild-config.yml in your project directory (i.e. along side guild.yml). This file provides the same config as ~/.guild/config.yml` but applies to your project rather than to your system.

  1. In 0.9, Guild copies source code to the run directory root, rather than to .guild/sourcecode. You can change this in your Guild file (guild.yml) using the sourcecode attribute for your operation.
# guild.yml

train:
  sourcecode:
    dest: .guild/sourcecode
    ...

However, in your case, consider changing exec simply to python train.py.

I apologize for the breakage here! Guild should have done a better job here at surfacing the problem and pointing you to a solution. I’ve opened an issue to address this in 0.9.1 with additional help.

We’ll also confirm that the legacy switch above is properly documented.

3

Thank you Garrett! I’ve managed to get my projects working again. Thank you for detailing the relevant changes here as well; it made what I was seeing a lot more clear.