气流1.10安装失败 [英] Airflow 1.10 Installation Failing

查看:87
本文介绍了气流1.10安装失败的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我在Amazon EC2实例上运行的使用Airflow版本1.9的工作环境正常.我需要升级到最新版本的Airflow,即1.10.我可以选择从1.9版升级,也可以在新服务器上全新安装1.10. Pip中未列出Airflow版本1.10,因此我是通过此命令从Git安装的,

I have a working Airflow environment using Airflow version 1.9 that is running on an Amazon EC2-Instance. I need to upgrade to the latest version of Airflow which is 1.10. I have the option of either upgrading from version 1.9 or installing 1.10 freshly on a new server. Airflow version 1.10 is not listed on Pip so I'm installing it from Git via this command,

pip-3.6 install git+git://github.com/apache/incubator-airflow.git@v1-10-stable

此命令成功安装了Airflow版本1.10.通过运行命令airflow version并查看输出,您可以看到

This command successfully installs Airflow version 1.10. You can see that by running the command airflow version and viewing the output,

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
   v1.10.0

当我尝试使用airflow scheduler启动Airflow Scheduler时,出现以下异常,

When I tried starting up the Airflow scheduler with airflow scheduler I get the following exception,

ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,195] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,195] {celery_executor.py:113} ERROR - No module named 'MySQLdb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 94, in sync
    state = task.state
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
    session = self.ResultSession()
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
    **self.engine_options)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 45, in create_session
    engine = self.get_engine(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 42, in get_engine
    return create_engine(dburi, poolclass=NullPool)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 110, in dbapi
    return __import__('MySQLdb')
ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,196] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,196] {celery_executor.py:113} ERROR - No module named 'MySQLdb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 94, in sync
    state = task.state
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
    session = self.ResultSession()
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
    **self.engine_options)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 45, in create_session
    engine = self.get_engine(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 42, in get_engine
    return create_engine(dburi, poolclass=NullPool)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 110, in dbapi
    return __import__('MySQLdb')
ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,197] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,197] {celery_executor.py:113} ERROR - No module named 'MySQLdb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 94, in sync
    state = task.state
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
    session = self.ResultSession()
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
    **self.engine_options)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 45, in create_session
    engine = self.get_engine(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 42, in get_engine
    return create_engine(dburi, poolclass=NullPool)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args^C[2018-08-14 14:03:16,424] {jobs.py:1585} INFO - Exited execute loop
[2018-08-14 14:03:16,433] {jobs.py:1599} INFO - Terminating child PID: 13615

这是我的lib文件夹中的内容,

Here's what my lib folder has,

[/usr/local/lib/python3.6/site-packages]# cd /usr/local/lib64/python3.6/site-packages/sqlalchemy/
root@ip-1-2-3-4
[/usr/local/lib64/python3.6/site-packages/sqlalchemy]# ll
total 320
drwxr-xr-x  3 root root  4096 Aug 13 17:17 connectors
-rwxr-xr-x  1 root root 40456 Aug 13 17:17 cprocessors.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root 51408 Aug 13 17:17 cresultproxy.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root 21944 Aug 13 17:17 cutils.cpython-36m-x86_64-linux-gnu.so
drwxr-xr-x  3 root root  4096 Aug 13 17:17 databases
drwxr-xr-x 10 root root  4096 Aug 13 17:17 dialects
drwxr-xr-x  3 root root  4096 Aug 13 17:17 engine
drwxr-xr-x  3 root root  4096 Aug 13 17:17 event
-rwxr-xr-x  1 root root 49746 Mar  6 14:01 events.py
-rwxr-xr-x  1 root root 12030 Mar  6 14:01 exc.py
drwxr-xr-x  4 root root  4096 Aug 13 17:17 ext
-rwxr-xr-x  1 root root  2249 Mar  6 14:01 __init__.py
-rwxr-xr-x  1 root root  3093 Mar  6 14:01 inspection.py
-rwxr-xr-x  1 root root 10967 Mar  6 14:01 interfaces.py
-rwxr-xr-x  1 root root  6712 Mar  6 14:01 log.py
drwxr-xr-x  3 root root  4096 Aug 13 17:17 orm
-rwxr-xr-x  1 root root 49883 Mar  6 14:01 pool.py
-rwxr-xr-x  1 root root  5217 Mar  6 14:01 processors.py
drwxr-xr-x  2 root root  4096 Aug 13 17:17 __pycache__
-rwxr-xr-x  1 root root  1200 Mar  6 14:01 schema.py
drwxr-xr-x  3 root root  4096 Aug 13 17:17 sql
drwxr-xr-x  5 root root  4096 Aug 13 17:17 testing
-rwxr-xr-x  1 root root  1713 Mar  6 14:01 types.py
drwxr-xr-x  3 root root  4096 Aug 13 17:17 util
root@ip-1-2-3-4
[/usr/local/lib64/python3.6/site-packages/sqlalchemy]# pwd
/usr/local/lib64/python3.6/site-packages/sqlalchemy
root@ip-1-2-3-4
[/usr/local/lib64/python3.6/site-packages/sqlalchemy]# cd /usr/local/lib/python3.6/site-packages/sqlalchemy/
bash: cd: /usr/local/lib/python3.6/site-packages/sqlalchemy/: No such file or directory

我只是感到困惑,为什么Airflow的安装未解决所有需要的依赖项.我是否正确安装了Airflow?我确实需要使用版本1.10,因为版本1.9中有一个主要错误,如此处此处. /p>

I'm just confused why Airflow's installation didn't take care of all it's needed dependencies. Am I installing Airflow incorrectly? I really need to be on version 1.10 because version 1.9 has a major bug in it as discovered here and here.

推荐答案

在进行全新安装时,可以提供许多安装额外功能(可选依赖项").默认情况下,Airflow不会全部安装它们,因为有数十种,有些需要特殊的依赖项,例如Mesos或Kubernetes.

There are a number of install extras ("optional dependencies") one can provide when doing a fresh install. Airflow doesn't install them all by default because there are dozens and some require special dependencies like Mesos or Kubernetes.

https://airflow.readthedocs.io/en/stable /installation.html#extra-packages

请注意,对于1.10.0-1.10.2,您现在需要在安装命令前添加序号或导出此环境变量:

Note that for 1.10.0-1.10.2 you now need to preface install commands or export this env var:

export SLUGIFY_USES_TEXT_UNIDECODE=yes

这不再是必需的

This is no longer required for 1.10.3 and up.

1.10发布后,您将可以安装以下附加功能:

Once 1.10 is released you'll be able to install extras like this:

pip install apache-airflow[celery,devel,postgres]

从git安装时,用于安装Extras的pip语法要复杂一些:

When installing from git, the pip syntax for installing extras is a little more complicated:

pip install git+git://github.com/apache/incubator-airflow.git@v1-10-stable#egg=apache-airflow[celery,devel,postgres]

如果您尝试安装具有MySQL支持的Airflow,则可以添加mysql extra:

If you're trying to install Airflow with MySQL support, you can include the mysql extra:

pip install git+git://github.com/apache/incubator-airflow.git@v1-10-stable#egg=apache-airflow[mysql]

如果您确实要安装所有附加功能,则可以使用all附加功能:

If you really do want to install all extras, you can use the all extra:

pip install git+git://github.com/apache/incubator-airflow.git@v1-10-stable#egg=apache-airflow[all]

注意:如果您以前在PyPI上安装了apache-airflow 1.9的任何其他功能,则从GitHub安装1.10时,您需要在此处再次提供它们,因为pip不会将GitHub存储库与PyPI软件包相关联.

Note: If you previously installed any extras for apache-airflow 1.9 on PyPI, you'd need to provide them again here when installing 1.10 from GitHub since pip doesn't associate the GitHub repo with the PyPI package.

问题

  • 您正在运行Python 3.6.5吗?
  • 如果您在安装中包含mysql Extra,仍然会遇到相同的错误吗?
  • Are you running Python 3.6.5?
  • Do you still get the same error if you include mysql extra on install?

这篇关于气流1.10安装失败的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆