在git版本之间删除或修改的文件会自动从实例中删除 [英] Files deleted or modified between git revisions are automatically getting deleted from instances

问题描述

背景

我有一个由詹金斯(Jenkins)触发的设置，其中包含以下内容-

I have a setup triggered by Jenkins with the following -

• 要部署的文件是通过phing，与git服务器通信并在单独的构建服务器中在所需的git版本之间获取git diff来准备的，而无需参与AWS代码部署(据我认为) . phing版本由Jenkins触发.
• 我仅将要添加/修改的文件(基于修订版的git差异)动态添加到appspec.yml文件.我只准备要添加/修改到路径/home/jenkins/deployment/cd_deploy/codebase/的文件，并且已经在Jenkins项目的高级项目选项"下的使用自定义工作区"选项下指定了路径/home/jenkins/deployment/cd_deploy/，这基本上是构建服务器中的位置需要将其上传到S3存储桶. 请注意，我需要从两个git版本之间删除的实例中删除文件.
• Jenkins然后使用有关应用程序名称，我已配置的代码部署的部署组的信息来触发AWS Codedeploy.

• The files to be deployed are prepared by phing, by talking to git server and taking a git diff between the required git revisions, in a separate build server, without the involvement of AWS code deploy (as far as I think). The phing build is triggered by Jenkins.
• I add only the files to be added/modified (based on the git difference of revisions) dynamically to the appspec.yml file. I prepare only the files to be added/modified to a path /home/jenkins/deployment/cd_deploy/codebase/ and I have specified the path /home/jenkins/deployment/cd_deploy/ under "Use custom workspace" option under "Advanced Project Options" of Jenkins project which is basically the location in the build server which needs to be uploaded to the S3 bucket. Note that I would need to delete the files from the instances that are deleted between the two git revisions.
• Jenkins then triggers AWS Codedeploy with the information about the application name, deployment group of Code deploy that I have configured.

问题

我动态地添加到appspec.yml文件中的文件正在EC2实例中被修改/添加，但是，奇怪的是，要删除的文件也被删除了.我确认我没有逻辑删除在我的appspec文件的beforeInstall钩子中写入的那些文件.我的beforeInstall挂钩中只有一个beforeInstall.sh文件，而没有其他挂钩.一旦我从appspec文件中删除了该钩子，删除就会停止.这是我的appspec文件-

The files that I dynamically add to the appspec.yml file are getting modified/added in the EC2 instances, as I expect, however, strangely, the files that are to be deleted are also getting deleted. I verified that I have no logic to delete those files written in the beforeInstall hook of my appspec file. I have only a beforeInstall.sh file in my beforeInstall hook, and no other hook. As soon as I remove that hook from the appspec file, the deletion stops. Here is my appspec file -

version: 0.0
os: linux
files:
{Pair of files dynamically generated}
  - source: config/deployment_config.json
    destination: /var/cake_1.2.0.6311-beta/deployment
permissions:
  - object: .
    pattern: "**"
    owner: sandeepan
    group: sandeepan
    mode: 777
    type:
      - file
hooks:
  BeforeInstall:
    - location: beforeInstall.sh

AWS Codedeploy是否以某种方式与我的git托管进行了交谈(我使用的是gitlab，甚至没有使用github)，并以某种方式获取有关要删除的文件的信息.

Is AWS Codedeploy somehow talking to my git hosting (I am using gitlab and not even github) and somehow getting the information about the files to be deleted.

更新

我后来发现，即使从appspec.yml文件中完全删除了hooks部分，并从中央构建服务器(准备了S3捆绑包的位置)中删除了相应的.sh文件，即beforeInstall.sh，afterInstall.sh等， )，因此我的逻辑及其引用均不涉及实例，要删除的文件仍会自动删除.

I later observed that even after removing the hooks section completely from the appspec.yml file, and deleting the corresponding .sh files, i.e. beforeInstall.sh, afterInstall.sh etc from the central build server (where the S3 bundle is prepared), so that none of my logic and any reference to it is going to the instances, the files that are to be deleted are still getting deleted automatically.

更新2

今天，我发现在git版本之间进行修改的文件也将被自动删除. 我具有动态准备appspec.yml文件的逻辑.我修改为不添加一些文件.因此，git diff中有一些文件，而appspec文件中没有这些文件.结果，它们将被删除但不会再次出现. 看来，代码部署在部署之前会自动进行清理.我该如何停止呢?我想添加我的自定义清理逻辑.

Today I found that the files that are modified in between git revisions are also getting deleted automatically. I had logic to dynamically prepare the appspec.yml file. I modified to not add some files. So, there were some files which were there in the git diff, but were not there in the appspec file. As a result, they are getting deleted but not reappearing. Code deploy is automatically doing a cleanup before the deployment, it seems. How do I stop that? I would like to add my custom cleanup logic.

更新3

beforeInstall.sh的内容-

Content of beforeInstall.sh -

OUTPUT="$(w | grep -Po '(?<=load average: )[^,]*')"
rm -f /var/cake_1.2.0.6311-beta/deployment/deployment_config.json
path="$PWD"
php $path"/deployment-root/"$DEPLOYMENT_GROUP_ID"/"$DEPLOYMENT_ID"/deployment-archive/beforeInstall.php" ${OUTPUT}

/usr/local/nagios/libexec/check_logwarn -d /tmp/logwarn_hiphop_error /mnt/log/hiphop/error_`(date +'%Y%m%d')`.log #Just run a nagios check, so that counter corresponds to the line in the log corresponding to current timestamp/instant. Do not care about output. Note that we are not even looking for error hinting keywords (and hence not using -p because it needs to be used alongwith), because all we need to care about here is incrementing the nginx counter.

/usr/local/nagios/libexec/check_logwarn -d /tmp/logwarn_nginx_access /mnt/log/nginx/access_`(date +'%Y%m%d')`_`(date +'%H')`.log #Just run a nagios check, so that counter corresponds to the line in the log corresponding to current timestamp/instant. Acceptable http codes are also not being read from deployment_config.json.
printf "\n `date +%Y-%m-%d:%H:%M:%S` End of beforeInstall.sh"  >> /var/cake_1.2.0.6311-beta/deployment/deployment.log
exit 0

以及从上面调用的beforeInstall.php的内容-

And content of beforeInstall.php which is called from the above -

<?php 
file_put_contents('/var/cake_1.2.0.6311-beta/deployment/deployment.log', "\n ".date("Y-m-d H:i:s")." - Load print  ".$argv[1], FILE_APPEND);
$loadData = json_encode(array("load" => intval($argv[1]), "access_error_check_day" => date("Ymd"), "access_error_check_hour" => date("H"))); //error_check_day -> day when nagios error check was last run. We will accordingly check log files of days in between this day and the day of afterinstall (practically this can include a span of 2 days).

file_put_contents("/var/cake_1.2.0.6311-beta/deployment/serverLoad.json",$loadData); //separate from deployment_config.json. serverLoad.json is not copied from build server.
file_put_contents('/var/cake_1.2.0.6311-beta/deployment/deployment.log', "\n ".date("Y-m-d H:i:s")." loadData to config ".$loadData, FILE_APPEND);
?>

推荐答案

CodeDeploy旨在部署应用程序，而不是简单地复制一组特定且不断变化的文件.

CodeDeploy is designed to deploy applications, not simply copy a specific and constantly different set of files.

这样，在部署每个修订"之前，CodeDeploy将首先清除由 previous 修订所部署的所有文件.让我解释一下.

As such, before deploying each 'revision', CodeDeploy will first cleanup any files deployed by the previous revision. Let me explain.

因此，假设先前的应用程序部署上传了三个文件:

So, let's say the previous app deployment uploaded three files:

File A
File B
File C

然后下一次部署仅包括以下文件:

And then the next deployment only included these files:

File A
File C

代码部署将首先清理它在第一个修订版(A，B和C)上部署的3个文件，然后部署新的修订版...它永远不会简单地上传预期的文件，始终首先清除旧文件(通过查看先前的修订"来确定).这很重要，因为它为您的案例中看起来神秘的行为提供了一些线索.部署后的结果当然是:

Code Deploy will first cleanup the 3 files it deployed on the first revision (A, B and C), and then deploy your new revision... It never simply uploads the files intended, it always cleans up the old files first (determined by looking at the previous 'revision'). This is important because it sheds some light on what seems like mysterious behavior in your case. The result, after deployment is, of course:

File A
File C

现在，如果您手动将文件添加到CodeDeploy之外的组合中，将变得很有趣.它只会清除其了解的内容，并且如果此清除阶段未删除文件，它也不会覆盖当前版本中的文件.当人们手动安装应用程序，然后尝试将CodeDeploy部署到同一文件夹时，通常会出现这种情况.没有以前的版本，因此无需清除任何内容，然后尝试在现有文件之上复制并会出错.通常，您希望目标文件夹是裸"的，以便您可以正确启动修订历史记录.

Now, it gets interesting if you've manually added files into the mix outside of CodeDeploy. It will only clean things it knows about, and it also won't overwrite files in the current revision if this cleanup phase doesn't remove them. This is often seen when people have manually installed an application, and then tried to do a CodeDeploy to the same folder... there's no previous revision, so nothing to clean up, and then it tries to copy on top of the existing files and will error out. You typically want your target folder to be 'naked' so you can start the revision history properly.

例如，在先前的情况下，如果您先前手动 上载了文件A，B和C，则部署文件A& B会失败，因为它不知道先清理A，B和C，然后在尝试覆盖文件A和B时给您一个错误.

For example, in that previous scenario, if you had previously uploaded the Files A, B and C manually, then the deployment of Files A & B would have failed because it wouldn't know to clean up A, B and C first, and it would then give you an error trying to overwrite the files A and B.

完全不在部署范围内的文件(或文件夹)...即不是任何修订版本的一部分，比如说文件D ...在部署之前和之后都不会受到影响并愉快地保留在那里，而无需部署抱怨.这对于放置数据文件以及某些可能特定于部署的东西很有用，但这些东西不一定是您不想不断重新部署的代码库的一部分.

A file (or folder) completely outside the deployment... i.e. not part of either revision, say File D... would be untouched and remain happily there both before and after the deployment without complaint. This is useful for placing data files and such things that may be specific to the deployment but aren't necessarily part of the code base that you don't want to constantly redeploy.

现在，您当然可以使用弯钩来做很多有趣的事情，但是感觉像是对手头工作的错误工具.这些挂钩旨在执行诸如停止/启动服务之类的操作，而不是管理CodeDeploy应该为您执行的工作的核心文件复制管理.

Now, you can do lots of interesting things using the hooks, of course, but it feels like the wrong tool for the job at hand. The hooks are intended for doing things like stop/start services, etc. not to manage the file copy management that is at the heart of what CodeDeploy should be doing for you.

从应用程序规范中排除所有文件(即未指定文件)，仅使用BeforeInstall和/或AfterInstall步骤执行复制逻辑是一种适用于某些情况的方法.

Excluding all files from the app spec (i.e. no files specified) and simply using BeforeInstall and/or AfterInstall steps to perform the copy logic is an approach that may work for some scenarios.

无论如何，对CodeDeploy的运行方式的更好理解也许可以帮助您制定解决方案.我认为它没有特别有据可查的.我的理解来自我自己的观察和奋斗.

In any case, maybe this better understanding of how CodeDeploy operates might help you craft a solution. I don't think its particularly well documented. My understanding comes from observing and struggling with it myself.

这篇关于在git版本之间删除或修改的文件会自动从实例中删除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！