如何在Flink中加载外部jar [英] How to load external jars in Flink

问题描述

当我以独立集群模式将作业提交给Flink时，我发现taskManager每次都会从jobManager中获取jar(即使对于同一个jar)，这也需要很长时间.我想知道是否有可能将这些jars保留在每个工作程序节点中，以便它们将在每次运行时自动在本地加载jars.

When I submit jobs to Flink in a standalone cluster mode, I find each time the taskManager will fetch the jar from the jobManager (even for the same jar), which takes a long time. I am wondering whether it is possible to keep these jars in each worker node such that they will automatically load the jars locally for each run.

推荐答案

您可以仅使用内部代码/逻辑创建最小的jar，并确保您依赖的所有jar在/lib中可用文件夹(根据Arthur的建议).这将使从JobManager提取jar所需的时间最少.此Flink文档说应该有可能(在某些情况下)避免加载任何动态jar，但我从未尝试过这样做.

You can create a minimal jar with just your internal code/logic, and ensure that all of the jars you depend on are available in the /lib folder (as per Arthur's suggestion). That will result in minimal time fetching the jar from the JobManager. This Flink documentation says it should be possible (in some situations) to avoid having to load any dynamic jars, but I've never tried that.

这篇关于如何在Flink中加载外部jar的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！