Spring Batch-并非从MQ检索中正在处理所有记录 [英] Spring Batch - Not all records are being processed from MQ retrieval

问题描述

我是Spring和Spring Batch的新手，请随时提出任何澄清的问题。

I am fairly new to Spring and Spring Batch, so feel free to ask any clarifying questions if you have any.

我看到了Spring Batch的一个问题我无法在测试或本地环境中重新创建。我们的日常工作是通过JMS连接到Websphere MQ，并检索一组记录。这项工作使用现成的JMS ItemReader。我们实现了自己的ItemProcessor，但是除了日志记录之外，它没有做任何其他特别的事情。没有过滤器或处理会影响传入的记录。

I am seeing an issue with Spring Batch that I cannot recreate in our test or local environments. We have a daily job that connects to Websphere MQ via JMS and retrieves a set of records. This job uses the out-of-the-box JMS ItemReader. We implement our own ItemProcessor, but it doesn't do anything special other than logging. There are no filters or processing that should affect incoming records.

问题在于，在MQ的10,000多个每日记录中，只有大约700个左右（确切的数目）每次都不同）通常会记录在ItemProcessor中。所有记录均已成功从队列中取出。每次记录的记录数都不相同，似乎没有规律。通过将日志文件与MQ中的记录列表进行比较，我们可以看到我们的作业正在处理一个看似随机的记录子集。第一条记录可能会被拾取，然后跳过50条，然后连续5条，依此类推。每次运行作业时，模式都不同。

The problem is that out of the 10,000+ daily records on MQ, only about 700 or so (the exact number is different each time) usually get logged in the ItemProcessor. All records are successfully pulled off the queue. The number of records logged is different each time and seems to have no pattern. By comparing the log files against the list of records in MQ, we can see that a seemingly random subset of records are being "processed" by our job. The first record might get picked up, then 50 are skipped, then 5 in a row, etc. And the pattern is different each time the job runs. No exceptions are logged either.

在localhost中运行相同的应用程序并使用相同的数据集进行测试时，ItemProcessor会成功检索并记录所有10,000多个记录。这项工作在生产阶段需要20到40秒（也不是恒定的），但是在测试和本地工作需要几分钟的时间才能完成（这显然很有意义，因为它正在处理更多的记录）。

When running the same app in localhost and test using the same data set, all 10,000+ records are successfully retrieved and logged by the ItemProcessor. The job runs between 20 and 40 seconds in Production (also not constant), but in test and local it takes several minutes to complete (which obviously makes sense since it is handling so many more records).

所以这是要解决的难题之一，因为我们无法重新创建它。一种想法是实现我们自己的ItemReader并添加其他日志记录，以便我们可以查看记录是在读取器之前还是在读取器之后丢失了-现在我们所知道的是ItemProcessor仅处理记录的一部分。但是即使那样也不能解决我们的问题，并且考虑到它甚至不是解决方案，实施该方案也将是适时的。

So this is one of those tough issue to troubleshoot since we cannot recreate it. One idea is to implement our own ItemReader and add additional logging so that we can see if records are getting lost before the reader or after the reader - all we know now is that only a subset of records are being handled by the ItemProcessor. But even that will not solve our problem, and it will be somewhat timely to implement considering it is not even a solution.

还有其他人看到过这样的问题吗？任何可能的想法或故障排除建议将不胜感激。这是一些供我们参考的jar版本号。

Has anyone else seen an issue like this? Any possible ideas or troubleshooting suggestions would be greatly appreciated. Here are some of the jar version numbers we are using for reference.

• Spring-3.0.5.RELEASE


• Spring Integration-2.0.3.Release


• Spring Batch-2.1.7.RELEASE


• Active MQ-5.4.2


• Websphere MQ-7.0.1

• Spring - 3.0.5.RELEASE
• Spring Integration - 2.0.3.RELEASE
• Spring Batch - 2.1.7.RELEASE
• Active MQ - 5.4.2
• Websphere MQ - 7.0.1

谢谢您的输入。

编辑：每个请求，处理器代码：

Per request, code for processor:

public SMSReminderRow process(Message message) throws Exception {

    SMSReminderRow retVal = new SMSReminderRow();
    LOGGER.debug("Converting JMS Message to ClaimNotification");
    ClaimNotification notification = createClaimNotificationFromMessage(message);

    retVal.setShortCode(BatchCommonUtils
            .parseShortCodeFromCorpEntCode(notification.getCorpEntCode()));
    retVal.setUuid(UUID.randomUUID().toString());
    retVal.setPhoneNumber(notification.getPhoneNumber());
    retVal.setMessageType(EventCode.SMS_CLAIMS_NOTIFY.toString());

    DCRContent content = tsContentHelper.getTSContent(Calendar
            .getInstance().getTime(),
            BatchCommonConstants.TS_TAG_CLAIMS_NOTIFY,
            BatchCommonConstants.TS_TAG_SMSTEXT_TYP);

    String claimsNotificationMessage = formatMessageToSend(content.getContent(),
            notification.getCorpEntCode());

    retVal.setMessageToSend(claimsNotificationMessage);
    retVal.setDateTimeToSend(TimeUtils
            .getGMTDateTimeStringForDate(new Date()));

    LOGGER.debug(
            "Finished processing claim notification for {}. Writing row to file.",
            notification.getPhoneNumber());
    return retVal;
}

JMS配置：

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:context="http://www.springframework.org/schema/context"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd">
<bean id="claimsQueueConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="jms/SMSClaimNotificationCF" />
    <property name="lookupOnStartup" value="true" />
    <property name="cache" value="true" />
    <property name="proxyInterface" value="javax.jms.ConnectionFactory" />
</bean>

<bean id="jmsDestinationResolver"
    class="org.springframework.jms.support.destination.DynamicDestinationResolver">
</bean>

<bean id="jmsJndiDestResolver" 
    class=" org.springframework.jms.support.destination.JndiDestinationResolver"/>  

<bean id="claimsJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="claimsQueueConnectionFactory" />
    <property name="defaultDestinationName" value="jms/SMSClaimNotificationQueue" />
    <property name="destinationResolver" ref="jmsJndiDestResolver" />
    <property name="pubSubDomain">
        <value>false</value>
    </property>
    <property name="receiveTimeout">
        <value>20000</value>
    </property>
</bean>

推荐答案

请参见 http://activemq.apache.org/jmstemplate-gotchas.html。

使用JMSTemplate存在问题。我只是在升级硬件并突然暴露先前存在的竞争状况时才遇到这些问题。

There are issues using the JMSTemplate. I only ran into these issues when I upgraded my hardware and suddenly exposed a pre-existing race condition.

简写形式是，JMS模板是根据设计和意图打开的，关闭每个调用上的连接。它不会看到早于其创建的消息。在高容量和/或高吞吐量的情况下，它将无法读取某些消息。

The short form is that by design and intent the JMS Template opens and closes the connection on every invocaton. It will not see messages older than its creation. In high volume and/or high throughput scenarios, it will fail to read some messages.

这篇关于Spring Batch-并非从MQ检索中正在处理所有记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！