使用git推送和拉我的conda环境 [英] Push and pull my conda environment using git

问题描述

我的项目中有一个git repo. 我经常更改conda环境，因此我希望我的仓库可以跟踪环境中的变化，并能够推送最新的并将其拉入另一台计算机.是否有可能? 我搜索并找到了几种解决方案(例如 https://tdhopper.com/blog/my-python-environment-workflow-with-conda/)，但没有一个提供自动的更改跟踪.

意思是，我要将我在环境中所做的任何更改都包含在项目的存储库中.就像添加新软件包等.这样，当我在另一台计算机上git pull它时，新软件包也将被提取并添加到环境中.

解决方案

我使用 git hooks 使conda环境自动更新.您可以在此处上获得有关git hooks的更多信息./p>

这里的想法是有两个git钩子:

• 一个用于检测本地conda环境是否发生更改的程序，如果发生更改，请使用更新的 env.yml 文件创建一个新提交(我选择了一个 pre-push >为此钩住.)
• 一个在拉出后检测到 env.yml 文件中的更改的文件(即，远程 env.yml 与本地文件不同，并且被合并了，我选择了一个合并后与此挂钩)

如文档中所述，当启动git存储库时，将创建一个文件夹 .git/hooks 并填充示例脚本.要使用其中之一，只需编辑文件，将其重命名以删除其扩展名( .sample )并确保其可执行文件即可.

注意:我使用zsh作为shell，但是bash中的脚本应该是相同的(如果没有，请注释)，您只需要更改shebang行即可.

---

预推钩

• 重写 .git/hooks 中已经存在的 pre-push.sample 文件(用conda环境的名称替换<ENV_NAME>):

 #!/usr/bin/env zsh

echo "\n==================== pre-push hook ===================="

# Export conda environment to yaml file
conda env export -n <ENV_NAME> env.yml

# Check if new environment file is different from original 
git diff --exit-code --quiet env.yml 

# If new environment file is different, commit it
if [[ $? -eq 0 ]]; then
    echo "Conda environment not changed. No additional commit."
else
    echo "Conda environment changed. Commiting new env.yml"
    git add env.yml
    git commit -m "Updating conda environment"
    echo 'You need to push again to push additional "Updating conda environment" commit.'
    exit 1
fi
 

• 删除其扩展名 .sample ，并在必要时使其可执行(chmod u+x pre-push)

---

合并后挂钩

• 我的 .git/hooks 文件夹中没有 post-merge.sample 钩子，因此我创建了一个 post-merge 文件，使用以下要点 https://gist.github.com/sindresorhus/7996717 作为模板:

 #!/usr/bin/env zsh

echo "\n==================== post-merge hook ===================="

changed_files="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD)"

check_run() {
    echo "$changed_files" | grep --quiet "$1" && eval "$2"
}

echo "Have to update the conda environment"
check_run env.yml "conda env update --file env.yml"
 

• 使其可执行(chmod u+x post-merge)

---

现在会发生什么?

• 在推送时，如果conda环境发生了变化，则会显示一条消息，提示您必须再次推送以使用更新的 env.yml
推送提交.
• 拉动时，如果拉出的 env.yml 与本地 env.yml 不同，conda将使用新拉出的 env.yml更新本地环境. .

---

限制

• 如果环境在本地发生更改，则可以看到更新后的 env.yml 不会自动推送到远程.我从这篇文章 git commit in pre-push hook 中获取了建议.
• 当前，拉动后conda环境的更新使用的是合并后挂钩.我不知道例如在rebase情况下如何处理.
• 这里没有git专家，也许有更适合这些任务的钩子.
• 我注意到 env.yml 中的prefix部分，该部分提供了到本地计算机上环境文件夹的路径.经过一些测试，一切似乎都运行良好，但是我不知道在各种机器上进行开发时是否会以某种方式造成冲突.

所以……欢迎大家提出评论，更正和改进意见！

I have a git repo with my project. I change my conda environment quite frequently, so I want my repo to track changes in the environment, and to be able to push the most recent one and pull it in another computer. Is it possible? I search and find several solutions (e.g. https://tdhopper.com/blog/my-python-environment-workflow-with-conda/) but none provide an automatic changes-tracking.

Meaning, I want to include any changes I make in my environment into the project's repository. Like adding new packages etc. So that when I git pull it in another computer, the new package will be also pulled and added to the environment.

解决方案

I use git hooks to make conda environment updates automatic. You can have more information on git hooks here.

The idea here is to have two git hooks:

• One which detects if a change in your local conda environment occured and if so, create a new commit with the updated env.yml file (I chose a pre-push hook for this one).
• One which detects a change in env.yml file after a pull (i.e. the remote env.yml was different than the local one and was merged, I chose a post-merge hook for this one)

As described in the documentation, when a git repository is initiated, a folder .git/hooks is created and filled with example scripts. To use one of them, you only have to edit the file, rename it to remove its extension (.sample) and make sure it is executable.

NOTE: I use zsh as shell but the script should be the same in bash (please comment if not), you would just need to change the shebang line.

---

pre-push hook

• Rewrite the pre-push.sample file already present in .git/hooks (replace <ENV_NAME> by the name of your conda environment):

#!/usr/bin/env zsh

echo "\n==================== pre-push hook ===================="

# Export conda environment to yaml file
conda env export -n <ENV_NAME> env.yml

# Check if new environment file is different from original 
git diff --exit-code --quiet env.yml 

# If new environment file is different, commit it
if [[ $? -eq 0 ]]; then
    echo "Conda environment not changed. No additional commit."
else
    echo "Conda environment changed. Commiting new env.yml"
    git add env.yml
    git commit -m "Updating conda environment"
    echo 'You need to push again to push additional "Updating conda environment" commit.'
    exit 1
fi

• Remove its extension .sample and make it executable if necessary (chmod u+x pre-push)

---

post-merge hook

• There were no post-merge.sample hook in my .git/hooks folder, so I created a file post-merge and used this gist https://gist.github.com/sindresorhus/7996717 as template:

#!/usr/bin/env zsh

echo "\n==================== post-merge hook ===================="

changed_files="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD)"

check_run() {
    echo "$changed_files" | grep --quiet "$1" && eval "$2"
}

echo "Have to update the conda environment"
check_run env.yml "conda env update --file env.yml"

• And make it executable (chmod u+x post-merge)

---

What will happen now ?

• When pushing, if the conda environment changed, a message will show that you have to push again to push the commit with the updated env.yml
• When pulling, if the pulled env.yml differs from the local env.yml, conda will update the local environment with the newly pulled env.yml.

---

Limitations

• In case the environment changed locally, you can see that the updated env.yml is not automatically pushed to remote. I took the advice from this post git commit in pre-push hook.
• Currently the updating of the conda environment after pull is using a post-merge hook. I don't know how this will be handled in case of rebase for example.
• No git expert here, maybe there is hooks better suited for these tasks.
• I noticed a prefix section in the env.yml which give the path to your environment folder on your local machine. After some test, everything seems to run fine but I don't know if this could somehow create conflicts when developing on various machines.

So ... comments, corrections and ideas of improvements are more than welcome !

这篇关于使用git推送和拉我的conda环境的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！