如何将一些外部jars库配置到flink docker容器 [英] how to configure some external jars library to the flink docker container

问题描述

我正在使用以下配置运行flink docker映像.

I am running a flink docker image with the following configuration.

version: '2.1'
services:
  jobmanager:
    build: .
    image: flink
    volumes:
      - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    hostname: "jobmanager"
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
  taskmanager:
    image: flink
    volumes:
    - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - jobmanager:jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

运行日志如下

    taskmanager_1  | 2020-10-11 10:34:03,714 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner      [] -  Classpath: 
  /opt/flink/lib/flink-csv-1.11.0.jar:
/opt/flink/lib/flink-json-1.11.0.jar:
/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:
/opt/flink/lib/flink-table-blink_2.12-1.11.0.jar:
/opt/flink/lib/flink-table_2.12-1.11.0.jar:
/opt/flink/lib/log4j-1.2-api-2.12.1.jar:
/opt/flink/lib/log4j-api-2.12.1.jar:
/opt/flink/lib/log4j-core-2.12.1.jar:
/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:
/opt/flink/lib/flink-dist_2.12-1.11.0.jar:::

从日志中可以看到一些默认库已加载到系统中，但是我想添加一些jar文件，例如flink-jdbc_2.11-1.9.0.jar，该文件位于我的本地文件系统中.

from the logs you can see some default libraries loaded into the system, but I want to add some jars like flink-jdbc_2.11-1.9.0.jar, which is in my local filesystem.

我的本​​地jar文件夹为/usr/local/lib/python3.7/sitepackages/pyflink/lib

my local jar folder is /usr/local/lib/python3.7/sitepackages/pyflink/lib

我尝试过

volumes:
  - /opt/flink/lib/:/usr/local/lib/python3.7/site-packages/pyflink/lib

错误是

ERROR: for keras-flask-deploy-webapp-master_jobmanager_1  Cannot start service jobmanager: Mounts denied: 
The path /opt/flink/lib is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.

我尝试了这个，但是还是没用

I tried this but still not working

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib

[ERROR] Flink distribution jar not found in /opt/flink/lib.
taskmanager_1  | [ERROR] The execution result is empty.

如何将jdbc jar库链接到flink的docker容器?

How would I link the jdbc jar library to the flink 's docker container?

推荐答案

在这种情况下，解决方法是将带有特定jar的卷附加到容器中的某个临时位置，并覆盖run命令以将附加文件复制到/opt/flink/lib flink类路径文件夹:

The workaround in this case can be to attach the volume with your specific jars to some temporary location in container and override the run command to copy the attached files into the /opt/flink/lib flink classpath folder:

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib-custom
command: cp -a /opt/flink/lib-custom/. /opt/flink/lib/ && jobmanager

我想说，如果您从那里运行应用程序，那么调整jobmanager容器就足够了，但是如果没有的话-在taskmanager中也重复同样的操作.

I would say tweaking jobmanager container would be enough if you run the application from there, but if not - repeat the same in taskmanager as well.

这篇关于如何将一些外部jars库配置到flink docker容器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！