bash zcat头会导致pipefail吗? [英] bash zcat head causes pipefail?

问题描述

set -eu 
VAR=$(zcat file.gz  |  head -n 12)

工作正常

set -eu   -o pipefail
VAR=$(zcat file.gz  |  head -n 12)

导致bash失败退出. 这是怎么引起管道失败的?

causes bash to exit with failure. How is this causing a pipefail?

请注意，file.gz包含数百万行(〜750 MB，已压缩).

Note that file.gz contains millions of lines (~ 750 MB, compressed).

推荐答案

请仔细考虑一下.

1. 您是在告诉外壳程序，如果有任何组件发生故障，则应将整个管道视为已发生故障.
2. 您要告诉zcat将其输出写入head.
3. 然后，您要告诉head在比12行长得多的输入流中读取12行后退出.

1. You're telling the shell that your entire pipeline should be considered to have failed if any component failed.
2. You're telling zcat to write its output to head.
3. Then you're telling head to exit after reading 12 lines, out of a much-longer-than-12-line input stream.

课程中的错误:zcat的目标管道很早关闭，无法成功编写输入文件的解压缩版本！它无法通过某种错误的方式知道这是由于用户意图引起的.

Of course you have an error: zcat has its destination pipeline closed early, and wasn't able to successfully write a decompressed version of your input file! It doesn't have any way of knowing that this was due to user intent, via something erroneous happening.

如果您正在使用zcat写入磁盘，但磁盘空间或网络流用完了，并且连接中断，则完全正确并且适合退出并显示状态为失败.这只是该规则的另一种情况.

If you were using zcat to write to a disk and it ran out of space, or to a network stream and there was a connection loss, it would be entirely correct and appropriate for it to exit with a status indicating a failure. This is simply another case of that rule.

由操作系统给出的zcat的特定错误是EPIPE，由write系统调用在以下情况下返回:试图写入未打开的管道可以通过任何过程进行读取.

The specific error which zcat is being given by the operating system is EPIPE, returned by the write syscall under the following condition: An attempt is made to write to a pipe that is not open for reading by any process.

退出head(此FIFO的唯一读取器)之后，对于对返回的管道 not 的输入端的任何写操作都将是一个错误.为了使zcat默默地忽略写入其输出的错误，从而能够生成不准确的输出流而又没有反映此事件的退出状态，同样， 就是一个错误.

After head (the only reader of this FIFO) has exited, for any write to the input side of pipeline not to return EPIPE would be a bug. For zcat to silently ignore an error writing its output, and thus be able to generate an inaccurate output stream without an exit status reflecting this event, would likewise be a bug.

如果您不想更改任何shell选项，那么您可能会考虑的一种解决方法是使用进程替换:

If you don't want to change any of your shell options, by the way, one workaround you might consider is using process substitution:

var=$(head -n 12 < <(zcat file.gz))

在这种情况下，zcat不是管道组件，并且出于确定成功的目的，不考虑其退出状态. (如果要提出独立的成功/失败判断标准，则可以测试$var是否为12行).

In this case, zcat is not a pipeline component, and its exit status is not considered for purposes of determining success. (You might test whether $var is 12 lines long, if you want to come up with an independent success/fail determination).

可以通过引入具有本地gzip支持的Python解释器来实现更全面的解决方案.嵌入到shell脚本中的本机Python实现(与Python 2和3.x兼容)可能类似于:

A more comprehensive solution could be implemented by pulling in a Python interpreter, with its native gzip support. A native Python implementation (compatible with both Python 2 and 3.x), embedded in a shell script, might look something like:

zhead_py=$(cat <<'EOF'
import sys, gzip
gzf = gzip.GzipFile(sys.argv[1], 'rb')
outFile = sys.stdout.buffer if hasattr(sys.stdout, 'buffer') else sys.stdout
numLines = 0
maxLines = int(sys.argv[2])
for line in gzf:
    if numLines >= maxLines:
        sys.exit(0)
    outFile.write(line)
    numLines += 1
EOF
)
zhead() { python -c "$zhead_py" "$@"; }

...这将为您提供zhead，如果它用完了输入数据也不会失败，但是确实会因为真正的I/O故障或其他意外情况而通过失败的退出状态事件. (用法的格式为zhead in.gz 5，可以从in.gz读取5行).

...which gets you a zhead that doesn't fail if it runs out of input data, but does pass through a failed exit status for genuine I/O failures or other unexpected events. (Usage is of the form zhead in.gz 5, to read 5 lines from in.gz).

这篇关于bash zcat头会导致pipefail吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！