如何在gitlab-ci脚本部分中使用while循环 [英] How to use while loop in gitlab-ci script section

问题描述

我正在尝试遍历文件中的url条目，并将每个文件用作爬网程序工具的输入.结果应写入文件.

I'm trying to iterate over url entries in a file and use each file as an input for a crawler tool. It's result should be written to a file.

这是gitlab-ci.yml文件:

here is the gitlab-ci.yml file:

stages:
  - test
test:
  stage: test

  tags:
    - shell-docker
  script:
    - wget https://github.com/FaKeller/sireg/releases/download/v0.3.1/sireg-linux
    - chmod 775 sireg-linux
    - mkdir output
    - ls -alF
    - while read line; do
         echo $line;
         ./sireg-linux exec --loader-sitemap-sitemap \"$line\" >> ./output/${line##*/}_out.txt;
      done < sitemap-index
    - ls -alF output
  artifacts:
    paths:
      - output/*
    expire_in: 1 hrs

这是站点地图索引文件(只有一项):

and here is the sitemap-index file (only one entry):

http://example.com/sitemap.xml

两个文件都在同一目录中.我希望将文件sitemap.xml_out.txt写入输出文件夹(也是同一目录).我非常确定./sireg-linux脚本不会执行，因为它通常需要几分钟才能完成(在本地测试).

both files are in the same directory. I expect a file sitemap.xml_out.txt to be written into the output folder(also the same directory). I am pretty sure the ./sireg-linux script does not execute because it usually takes few minutes to complete (tested locally).

阶段的输出如下:

2020-04-02 18:22:21 (4,26 MB/s) - »sireg-linux« saved [62566347/62566347]

$ chmod 775 sireg-linux
$ mkdir output
$ ls -alF
total 61128
drwxrwxr-x  4 gitlab-runner gitlab-runner     4096 Apr  2 18:22 ./
drwxrwxr-x 10 gitlab-runner gitlab-runner     4096 Apr  2 15:46 ../
drwxrwxr-x  5 gitlab-runner gitlab-runner     4096 Apr  2 18:22 .git/
-rw-rw-r--  1 gitlab-runner gitlab-runner      512 Apr  2 18:22 .gitlab-ci.yml
drwxrwxr-x  2 gitlab-runner gitlab-runner     4096 Apr  2 18:22 output/
-rw-rw-r--  1 gitlab-runner gitlab-runner       30 Apr  2 15:46 README.md
-rwxrwxr-x  1 gitlab-runner gitlab-runner 62566347 Nov 11  2017 sireg-linux*
-rw-rw-r--  1 gitlab-runner gitlab-runner       55 Apr  2 18:08 sitemap-index
$ while read line; do echo $line; ./sireg-linux **exec** --loader-sitemap-sitemap \"$line\" >> 
./output/${line##*/}_out.txt; done < sitemap-index
$ ls -alF output
total 8
drwxrwxr-x 2 gitlab-runner gitlab-runner 4096 Apr  2 18:22 ./
drwxrwxr-x 4 gitlab-runner gitlab-runner 4096 Apr  2 18:22 ../
Uploading artifacts...
Runtime platform                                    arch=amd64 os=linux pid=23813 revision=1f513601 version=11.10.1
WARNING: output/*: no matching files               
ERROR: No files to upload                          
Job succeeded

更新

试图将所有步骤移动到一个单独的脚本中，但这也不起作用.

tried to move all steps into a separate script but that did not work either.

更新2

忘记在命令中添加exec:

forgot to add exec in the command:

./sireg-linux exec --loader-sitemap-sitemap \"$line\" >> 
./output/${line##*/}_out.txt;

不幸的是，它没有帮助.

unfortunately it didn't help.

我该怎么做才能使其正常工作?

what can I do to make it working?

推荐答案

尝试将 ./sireg-linux --loader-sitemap-sitemap \"$ line \" 更改为./sireg-linux exec --loader-sitemap-sitemap"$ line" .希望这会有所帮助！

Try changing ./sireg-linux --loader-sitemap-sitemap \"$line\" to ./sireg-linux exec --loader-sitemap-sitemap "$line". Hope this helps!

此外，脚本似乎根本没有进入while循环.也许文件sitemap-index为空，或者只有一行，最后没有换行符?

Also, it looks like the script doesn't enter the while loop at all. Maybe the file sitemap-index is empty or it has only one line without a newline at the end?

命令行中的反斜杠是错误的.更正了我的答案

EDIT 2: The back-slashes in the command line are wrong. corrected my answer

这篇关于如何在gitlab-ci脚本部分中使用while循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！