如何为 flink docker 容器配置一些外部 jars 库 [英] how to configure some external jars library to the flink docker container

问题描述

我正在使用以下配置运行 flink docker 镜像.

I am running a flink docker image with the following configuration.

version: '2.1'
services:
  jobmanager:
    build: .
    image: flink
    volumes:
      - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    hostname: "jobmanager"
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
  taskmanager:
    image: flink
    volumes:
    - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - jobmanager:jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

运行日志如下

    taskmanager_1  | 2020-10-11 10:34:03,714 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner      [] -  Classpath: 
  /opt/flink/lib/flink-csv-1.11.0.jar:
/opt/flink/lib/flink-json-1.11.0.jar:
/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:
/opt/flink/lib/flink-table-blink_2.12-1.11.0.jar:
/opt/flink/lib/flink-table_2.12-1.11.0.jar:
/opt/flink/lib/log4j-1.2-api-2.12.1.jar:
/opt/flink/lib/log4j-api-2.12.1.jar:
/opt/flink/lib/log4j-core-2.12.1.jar:
/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:
/opt/flink/lib/flink-dist_2.12-1.11.0.jar:::

从日志中您可以看到一些默认库加载到系统中，但我想添加一些 jar，例如 flink-jdbc_2.11-1.9.0.jar，它在我的本地文件系统中.

from the logs you can see some default libraries loaded into the system, but I want to add some jars like flink-jdbc_2.11-1.9.0.jar, which is in my local filesystem.

我的本​​地 jar 文件夹是 /usr/local/lib/python3.7/sitepackages/pyflink/lib

my local jar folder is /usr/local/lib/python3.7/sitepackages/pyflink/lib

我试过了

volumes:
  - /opt/flink/lib/:/usr/local/lib/python3.7/site-packages/pyflink/lib

错误是

ERROR: for keras-flask-deploy-webapp-master_jobmanager_1  Cannot start service jobmanager: Mounts denied: 
The path /opt/flink/lib is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.

我试过了，但还是不行

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib

[ERROR] Flink distribution jar not found in /opt/flink/lib.
taskmanager_1  | [ERROR] The execution result is empty.

如何将 jdbc jar 库链接到 flink 的 docker 容器?

How would I link the jdbc jar library to the flink 's docker container?

推荐答案

这种情况下的解决方法可以是将带有特定 jar 的卷附加到容器中的某个临时位置，并覆盖运行命令以将附加文件复制到/opt/flink/lib flink 类路径文件夹:

The workaround in this case can be to attach the volume with your specific jars to some temporary location in container and override the run command to copy the attached files into the /opt/flink/lib flink classpath folder:

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib-custom
command: cp -a /opt/flink/lib-custom/. /opt/flink/lib/ && jobmanager

如果您从那里运行应用程序，我会说调整 jobmanager 容器就足够了，但如果不是 - 在 taskmanager 中也重复相同的操作.

I would say tweaking jobmanager container would be enough if you run the application from there, but if not - repeat the same in taskmanager as well.

这篇关于如何为 flink docker 容器配置一些外部 jars 库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！