数据流因错误而失败:找不到满足要求的版本(来自版本:无) [英] Dataflow fails due to ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)

问题描述

自今年年初以来，我们的 Python 数据流作业导致工作器启动时出错:

Since the beginning of this year our python dataflow jobs result in an error on worker startup:

Processing /var/opt/google/dataflow/setuptools-51.1.1.tar.gz
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
ERROR: No matching distribution found for wheel
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rvfx0hn9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the logs for full command output."

我的requirements.txt是

My requirements.txt is

ndjson==0.3.1
apache-beam[gcp]==2.26.0
dataclasses-json==0.5.2
google-cloud-storage==1.35.0
google-cloud-secret-manager==2.0.0

我正在使用以下 Dockerfile 创建一个 Flex 模板:

I am using the following Dockerfile to create a Flex-Template:

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}

# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
#   https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev git && rm -rf /var/lib/apt/lists/*

ADD utils utils
COPY requirements.txt .
COPY main.py .

ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"

RUN pip install -U apache-beam[gcp]==2.24.0
RUN pip install -U -r ./requirements.txt

有没有人解决这个问题?我确实尝试按照 以下帖子导致相同的错误消息.有没有人遇到过同样的问题

Has anyone a solution for the issue? I did try to set a specific setuptools version (51.1.1) as suggested in the following post leading to the same error message. Has anyone faced the same issue before

推荐答案

该问题是由于 dataclasses-json 中的冲突(我找不到确切原因).从 requirements.txt 中删除后，可以成功构建镜像:

The issue was due a conflict in the dataclasses-json (The exact reason I couldn't find out). After removing it from the requirements.txt the image can successfully be buildt:

ndjson==0.3.1
google-cloud-storage==1.31.2
google-cloud-secret-manager==2.0.0
pycryptodome==3.9.8

这篇关于数据流因错误而失败:找不到满足要求的版本(来自版本:无)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！