如何正确编译/打包 Spring Cloud Data Flow 的任务 [英] How to properly compile/package a Task for Spring Cloud Data Flow
问题描述
鉴于我需要使用多个数据源，我在 IntelliJ Idea 2020.1.3 中从 Spring Cloud 任务示例编译了以下示例:
当我尝试从仪表板运行任务时，执行实例没有提及任何开始时间或结束时间.它也不会在仪表板中显示任何内容.
使用任务执行日志从SCDF shell查询日志，显示很多错误.这似乎是最相关的部分:
2020-07-14 02:38:14.403 INFO 63 --- [ main] i.spring.MultipleDataSourcesApplication : 在 5856acfa7c62 上启动 MultipleDataSourcesApplication v2.3.0-SNAPSHOT，PID 为 63 (/root/scdf/multiple-datasources-2.3.0-SNAPSHOT.jar 由 root 启动，位于/tmp/289541567048/multiple-datasources-9c75a131-4ea9-40ff-ac42-44729162e6f5)2020-07-14 02:38:14.407 INFO 63 --- [main] i.spring.MultipleDataSourcesApplication:未设置活动配置文件，回退到默认配置文件:默认2020-07-14 02:38:17.242 INFO 63 --- [main] osjdeEmbeddedDatabaseFactory:启动嵌入式数据库:url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'2020-07-14 02:38:18.145 INFO 63 --- [main] o.s.j.d.e.EmbeddedDatabaseFactory :启动嵌入式数据库:url='jdbc:hsqldb:mem:testdb', username='sa'2020-07-14 02:38:18.810 调试 63 --- [主要] o.s.c.t.c.SimpleTaskAutoConfiguration:使用 io.spring.configuration.CustomTaskConfigurer TaskConfigurer2020-07-14 02:38:18.823 DEBUG 63 --- [main] o.s.c.t.c.DefaultTaskConfigurer:未找到 EntityManager，使用 DataSourceTransactionManager2020-07-14 02:38:18.928 DEBUG 63 --- [main] o.s.c.t.r.s.TaskRepositoryInitializer:初始化 h2 数据库的任务模式2020-07-14 02:38:19.036 ERROR 63 --- [ main] o.s.c.t.listener.TaskLifecycleListener :尚未开始的任务已收到结束任务的事件.2020-07-14 02:38:19.036 WARN 63 --- [main] s.c.a.AnnotationConfigApplicationContext:上下文初始化期间遇到异常 - 取消刷新尝试:org.springframework.context.ApplicationContextException:无法启动 bean 'taskLifecycleListener'；嵌套异常是 java.lang.IllegalArgumentException: Invalid TaskExecution, ID 31 not found2020-07-14 02:38:19.036 INFO 63 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory :关闭嵌入式数据库:url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false2020-07-14 02:38:19.245 INFO 63 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory :关闭嵌入式数据库:url='jdbc:hsqldb:mem:testdb'2020-07-14 02:38:19.258 ERROR 63 --- [ main] o.s.c.t.listener.TaskLifecycleListener :已收到尚未启动的任务的结束任务的事件.2020 年 7 月 14 日 02:38:19.264 信息 63 --- [主要] ConditionEvaluationReportLoggingListener :启动 ApplicationContext 时出错.要显示条件报告，请在启用调试"的情况下重新运行您的应用程序.2020-07-14 02:38:19.273 错误 63 --- [main] o.s.boot.SpringApplication:应用程序运行失败以下错误对我来说特别突出:
2020-07-14 02:38:19.036 WARN 63 --- [main] scaAnnotationConfigApplicationContext:上下文初始化期间遇到异常 - 取消刷新尝试:org.springframework.context.ApplicationContextException:无法启动 bean 'taskLifecycleListener';嵌套异常是 java.lang.IllegalArgumentException: Invalid TaskExecution, ID 31 not found2020-07-14 02:38:19.258 ERROR 63 --- [ main] o.s.c.t.listener.TaskLifecycleListener :已收到尚未启动的任务的结束任务的事件.2020 年 7 月 14 日 02:38:19.264 信息 63 --- [主要] ConditionEvaluationReportLoggingListener :启动 ApplicationContext 时出错.要显示条件报告，请在启用调试"的情况下重新运行您的应用程序.该示例在 application.properties 中有以下几行:
spring.application.name=Demo 多数据源任务logging.level.org.springframework.cloud.task=DEBUG`所以如果我没记错的话，调试应该已经启用了.
我的具体问题是:
1) 鉴于这是一个示例，即使不进行修改也无法运行，我可能会忽略或做错什么?
2) 如何正确启用 DEBUG?
谢谢
PS:repo 中的示例已经将 H2 数据库依赖项显示为 在其 pom.xml 中使用 Spring Cloud Data Flow 注册自定义 Spring Cloud 任务.
我没有尝试使用当前的启动初始化器重新创建示例，但是我尝试使用最新的初始化器创建一个简单的 hello-world，我得到了完全相同的错误.我不认为错误是初始化.
我还没有尝试最后一个建议，压倒一切.但鉴于这是官方示例，我真的需要覆盖默认配置吗?
PPS:我知道我的 SCDF 安装工作正常，因为我能够运行以下示例中的预打包时间戳程序:https://cloud.spring.io/spring-cloud-task-app-starters/
我遇到了同样的问题:每当我开始批处理作业时，TaskExecution 都会增加.
我查询了数据库(我在批处理作业中使用的)中的任务执行表，但没有发现与该执行 ID 相关的任何内容.
我尝试删除批处理作业并从 Docker Registry 重新导入应用程序，但任务执行仍然增加，没有希望！
我查看了用于在 Kubernetes 上安装 SCDF 的 SCDF server-config.yaml 文件，发现 SCDF 使用 Mysql 系统数据库来存储它的东西，这很奇怪！！
url: jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT}/mysql我查询了mysql系统数据库，在里面找到了带有ID信息的Task Execution Table！
问题解决，SCDF 本身将任务执行 ID 保存在自己的数据库中，而我们的批处理应用程序使用不同的数据库，因此它会产生问题.我必须临时将批处理应用程序数据库更改为 SCDF 的数据库，即 mysql 系统数据库.
来自:app.batch.spring.datasource.url=jdbc:mysql://mysql:3306/task?useSSL=false
致:app.batch.spring.datasource.url=jdbc:mysql://mysql:3306/mysql?useSSL=false
结论:SCDF 需要自己的数据库来管理任务执行，您的批处理应用程序(除了批处理的数据库)应该指向该 SCDF 的数据库才能工作.另外，在Kubernetes中安装SCDF的过程中，你应该将SCDF的数据库改为其他地方而不是mysql系统数据库
本教程将向您展示如何让您的批处理应用程序同时处理 2 个数据源
此链接将添加有关该问题的更多信息:
https://github.com/spring-io/dataflow.spring.io/issues/161
背景:
我按照这个链接安装了 SCDF 服务器(这让我发疯了)
https://dataflow.spring.io/docs/installation/kubernetes/kubectl/#deploy-data-flow-server
指向
还有这个链接
https://dataflow.spring.io/docs/batch-developer-guides/batch/spring-batch/
使用 mysql url 制作批处理应用程序的指南 url=jdbc:mysql://localhost:3306/task?useSSL=false (task database) 但它没有告诉你 SCDF 如何使用任务执行 ID
我想，Spring 应该更新文档并添加更多关于 SCDF 任务执行过程的解释
I compiled the following example in IntelliJ Idea 2020.1.3 from the Spring Cloud Task Samples, given that I need to use multiple datasources: https://github.com/mminella/spring-cloud-task/tree/master/spring-cloud-task-samples/multiple-datasources
I then packaged it in a JAR using mvn package, copied it to the location where I launched docker-compose (as per official SCDF (Spring Cloud Data Flow) instructions for local deployment) and added it to SCDF running the following command inside the SCDF shell: app register --type task --name multiple-datasources --uri file://root/scdf/multiple-datasources-2.3.0-SNAPSHOT.jar.
I added the task using the SCDF as shown:
When I try to run the task from the dashboard, the execution instance does not mention any start-time or end-time. Nor does it show any in the dashboard.
Consulting the log from the SCDF shell by using task execution log <instance>, many errors are shown. This seems like the most relevant part:
2020-07-14 02:38:14.403 INFO 63 --- [ main] i.spring.MultipleDataSourcesApplication : Starting MultipleDataSourcesApplication v2.3.0-SNAPSHOT on 5856acfa7c62 with PID 63 (/root/scdf/multiple-datasources-2.3.0-SNAPSHOT.jar started by root in /tmp/289541567048/multiple-datasources-9c75a131-4ea9-40ff-ac42-44729162e6f5)
2020-07-14 02:38:14.407 INFO 63 --- [ main] i.spring.MultipleDataSourcesApplication : No active profile set, falling back to default profiles: default
2020-07-14 02:38:17.242 INFO 63 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
2020-07-14 02:38:18.145 INFO 63 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:hsqldb:mem:testdb', username='sa'
2020-07-14 02:38:18.810 DEBUG 63 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using io.spring.configuration.CustomTaskConfigurer TaskConfigurer
2020-07-14 02:38:18.823 DEBUG 63 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : No EntityManager was found, using DataSourceTransactionManager
2020-07-14 02:38:18.928 DEBUG 63 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for h2 database
2020-07-14 02:38:19.036 ERROR 63 --- [ main] o.s.c.t.listener.TaskLifecycleListener : An event to end a task has been received for a task that has not yet started.
2020-07-14 02:38:19.036 WARN 63 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgumentException: Invalid TaskExecution, ID 31 not found
2020-07-14 02:38:19.036 INFO 63 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Shutting down embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false'
2020-07-14 02:38:19.245 INFO 63 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Shutting down embedded database: url='jdbc:hsqldb:mem:testdb'
2020-07-14 02:38:19.258 ERROR 63 --- [ main] o.s.c.t.listener.TaskLifecycleListener : An event to end a task has been received for a task that has not yet started.
2020-07-14 02:38:19.264 INFO 63 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-07-14 02:38:19.273 ERROR 63 --- [ main] o.s.boot.SpringApplication : Application run failed
The following errors in particular stand out to me:
2020-07-14 02:38:19.036 WARN 63 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgumentException: Invalid TaskExecution, ID 31 not found
2020-07-14 02:38:19.258 ERROR 63 --- [ main] o.s.c.t.listener.TaskLifecycleListener : An event to end a task has been received for a task that has not yet started.
2020-07-14 02:38:19.264 INFO 63 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
The example has the following lines in application.properties:
spring.application.name=Demo Multiple DataSources Task
logging.level.org.springframework.cloud.task=DEBUG`
so if I'm not mistaken, debug should already be enabled.
Concretely my questions are:
1) What might I be overlooking or doing wrong, given that this is an example and it's not running even without modifications?
2) What can I do to properly enable DEBUG?
Thank you
PS: The example from the repo already has the H2 database dependency shown as the answer in Registering Custom Spring Cloud Task with Spring Cloud Data Flow in its pom.xml .
I have not tried to re-create the example with a current boot initializr, however I have tried to make a simple hello-world with the most recent initializr and I get the exact same error. I do not think the error is the initializr.
I have yet to try the last suggestion, overriding. But given that this is an official example, should I really need to override the default configuration?
PPS: I know my installation of SCDF is working properly because I was able to run a pre-packaged timestamp programme from the example at: https://cloud.spring.io/spring-cloud-task-app-starters/
I got the same issue: whenever I start the batch job, the TaskExecution gets increased.
I queried the Task Execution Table in the database (that I use in my batch job) and found nothing related to that Execution ID.
I tried to delete the batch job and re import the application from Docker Registry and the Task Execution still gets increased, no hope!
I reviewed the SCDF server-config.yaml file, that were used for installing the SCDF on Kubernetes and found out that the SCDF used Mysql system database for storing its stuffs, it's weird!!
url: jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT}/mysql
I queried the mysql system database and found in there the Task Execution Table with ID information!
Problem solved, the SCDF itself saves Task Execution ID in its own database, and our batch application uses different database, so for that reason it makes the issue. I have to temporary change batch application database to SCDF's database, that's mysql system database.
From: app.batch.spring.datasource.url=jdbc:mysql://mysql:3306/task?useSSL=false
To: app.batch.spring.datasource.url=jdbc:mysql://mysql:3306/mysql?useSSL=false
Conclusion: SCDF needs its own database for managing Task Execution, and your batch application (beside batch's database) should point to that SCDF's database in order to work. Also, during the installation of the SCDF in Kubernetes, you should change the SCDF's database to somewhere else instead of mysql system database
This tutorial shows you how to make your batch application works with 2 datasources at the same time
This link will add more information on the issue:
https://github.com/spring-io/dataflow.spring.io/issues/161
Background:
I followed this link to install the SCDF server (that drove me crazy)
https://dataflow.spring.io/docs/installation/kubernetes/kubectl/#deploy-data-flow-server
which points to
And this link
https://dataflow.spring.io/docs/batch-developer-guides/batch/spring-batch/
guides to make batch application with the mysql url url=jdbc:mysql://localhost:3306/task?useSSL=false (task database) But it does not tell you how SCDF works with Task Execution ID
I guess, Spring should update the document and add more explanation on the SCDF Task Execution process
这篇关于如何正确编译/打包 Spring Cloud Data Flow 的任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
