为什么CXF / JAXB在编组到SOAP消息之前将整个InputStream读入内存 [英] Why CXF / JAXB read whole InputStream into memory before marshalling to SOAP message

问题描述

信息 - 示例代码

我已为您设置示例代码（SSCCE）以帮助您跟踪问题：

https://github.com/ ljader / test-cxf-base64-marshall

问题

<我正在与第三方JAX-WS服务集成，因此我无法更改 WSDL。

第三方Web服务期望 Base64 编码的字节对它们执行某些操作 - 它们期望客户端在SOAP消息中发送整个字节。
他们不希望更改为MTOM / XOP，因此我遇到了当前的要求。

我决定使用CXF轻松设置示例客户端，它适用于小文件。

但是当我尝试发送大数据，即200MB时，CXF / JAXB抛出一个异常：

 线程main中的异常java.lang.OutOfMemoryError：com.sun中的Java堆空间
。 xml.bind.v2.util.ByteArrayOutputStreamEx.readFrom（ByteArrayOutputStreamEx.java:75）
 at com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data.get（Base64Data.java:196) 
 at com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data.writeTo（Base64Data.java:312）
 at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.text（UTF8XmlOutput .java：312）
 at com.sun.xml.bind.v2.runtime.XMLSerializer.leafElement（XMLSerializer.java:356）
 at com.sun.xml.bind.v2.model.impl .RuntimeBuiltinLeafInfoImpl $ PcdataImpl.writeLeafElement（RuntimeBuiltinLeafInfoImpl.java:191）com.sun.x的
 ml.bind.v2.runtime.MimeTypedTransducer.writeLeafElement（MimeTypedTransducer.java:96）
 at com.sun.xml.bind.v2.runtime.reflect.TransducedAccessor $ CompositeTransducedAccessorImpl.writeLeafElement（TransducedAccessor.java:254）
 at com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty.serializeBody（SingleElementLeafProperty.java:130）
 at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody（ClassBeanInfoImpl .java：360）
 at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType（XMLSerializer.java:696）
 at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl $ 1.serializeBody（ElementBeanInfoImpl.java:155）
 at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl $ 1.serializeBody（ElementBeanInfoImpl.java:130）
 at com.sun.xml.bind .v2.runtime.ElementBeanInfoImpl.serializeBody（ElementBeanInfoImpl.java:332）
 at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeRoot（ElementBeanInfoImpl.java:339）
 at com.sun .xml.bind.v2.runtime.ElementBeanInfoImpl.serializ eRoot（ElementBeanInfoImpl.java:75）
 at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot（XMLSerializer.java:494）
 at com.sun.xml.bind.v2。 runtime.MarshallerImpl.write（MarshallerImpl.java:323）
 at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal（MarshallerImpl.java:251）
 at javax.xml.bind。 helpers.AbstractMarshallerImpl.marshal（AbstractMarshallerImpl.java:95）
 at org.apache.cxf.jaxb.JAXBEncoderDecoder.writeObject（JAXBEncoderDecoder.java:617）
 at org.apache.cxf.jaxb.JAXBEncoderDecoder。 marshall（JAXBEncoderDecoder.java:241）
 at org.apache.cxf.jaxb.io.DataWriterImpl.write（DataWriterImpl.java:237）
 at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts（在Org.apache.cxf.wsdl.interceptors.BareOutInterceptor.handleMessage（BareOutInterceptor.java:68）的
 org.apache.cxf.phase.PhaseInterceptorChain.doIntercept（PhaseInterceptorChain。）中的
。$ Abstract $。 java：308）
 at org.apache.cxf.endpoint.ClientImpl.doInv oke（ClientImpl.java:514）
 at org.apache.cxf.endpoint.ClientImpl.invoke（ClientImpl.java:423）
 at org.apache.cxf.endpoint.ClientImpl.invoke（ClientImpl。 java：324）
 at org.apache.cxf.endpoint.ClientImpl.invoke（ClientImpl.java:277）
 at org.apache.cxf.frontend.ClientProxy.invokeSync（ClientProxy.java:96） 
 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke（JaxWsClientProxy.java:139）
  

我的发现

我跟踪了基于xsd类型base64Binary的错误，

com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl

决定，

com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data

应处理来自

javax.activation.DataHandler

javax.activation.DataHandler

的数据编组p>

在编组期间，来自联合国的WHOLE数据尝试读取正在尝试的输入 http://grepcode.com/file/repo1.maven.org/maven2/com.sun.xml.bind/ jaxb-impl / 2.2.11 / com / sun / xml / bind / v2 / runtime / unmarshaller / Base64Data.java /＃311 ，导致OOME异常。

问题

CXF在将Java对象编组到SOAP消息期间使用JAXB - 当编组InputStream时，整个在转换为Base64二进制文件之前，输入流被读入内存。

所以我想以块发送（流）客户端到服务器的数据strong>（因为marshaller中的OutputSteam直接包装了HttpURLConnection），所以我的客户端可以处理发送任何数量的数据。

特别是当许多线程将使用我的客户端时，流媒体是恕我直言非常desira ble。

我没有很好的JAX-WS / CXF / JAXB知识，因此问题。

我发现并且可能有用的唯一材料是：

JAXB可以分块解析大型XML文件

http://rezarahim.blogspot.com/2010/05/chunking-out -big-xml-with-stax-and-jaxb.html

问题

1. 为什么CXF / JAXB将整个InputStream加载到内存中 - 不是DataHandler purpouse来阻止这样的实现？




2. 你知道如何将JAXB行为更改为不同的输入流吗？




3. 你知道不同的marshallers，它可以处理这样的大数据编组？




4. 作为最后的手段，也许你有链接到一些材料，如何创建自定义marshaller，将数据直接流到服务器？

解决方案

您不需要任何自定义marshallers或更改JAXB行为来实现您的需求 - DataHandler 是你的朋友。

回答你的第一个问题：JAXB需要将所有数据保存在内存中因为它必须解析引用。

我知道你无法更改WSDL引用等。但是你仍然在项目中拥有客户端的WSDL以生成客户课，不是吗？那么你可以做什么（我没有用第三方的WSDL测试 但可能值得尝试）是添加 xmime：expectedContentTypes =application / octet-stream进入响应XSD元素，返回Base64编码数据。例如：

 < xsd：element name =generateBigDataResponse> 
< xsd：complexType> 
< xsd：sequence> 
< xsd：element name =result
 type =xsd：base64Binary
 minOccurs =0
 maxOccurs =1
 xmime：expectedContentTypes = 应用/八位字节流/> 
< / xsd：sequence> 
< / xsd：complexType> 
< / xsd：element> 
  

另外不要忘记添加命名空间： xmlns：xmime =http： //www.w3.org/2005/05/xmlmime在 xsd：schema 元素中。

你在这里做什么 - 不改变任何WSDL引用，只是告诉JAXB而不是生成 byte [] 来生成 DataHandler 。那么当您生成类似的客户端类时会发生什么：

  @Override 
 public DataHandler generateBigData（）{
 try {
 final PipedOutputStream pipedOutputStream = new PipedOutputStream（）; 
 PipedInputStream pipedInputStream = new PipedInputStream（pipedOutputStream）; 
 InputStreamDataSource dataSource = new InputStreamDataSource（pipedInputStream，application / octet-stream）; 
 
 executor.execute（new Runnable（）{
 
 @Override 
 public void run（）{
 //将你的东西写入pipedOutputStream 
} 
}）; 
 
返回新的DataHandler（dataSource）; 
} catch（IOException e）{
 //处理异常，如果有任何
} 
} 
  

由于 xmime ，您获得 DataHandler 作为响应类型。我建议你使用 PipedOutputStream ，但要确保在另一个帖子中写作：

管道输出流可以连接到管道输入流到
创建通信管。管道输出流是管道的发送
端。通常，数据由一个线程写入PipedOutputStream
对象，并且由其他线程从连接的
PipedInputStream中读取数据。不建议尝试从单个线程使用两个对象
，因为它可能使线程死锁。
如果从连接的管道输入流中读取数据字节
的线程不再存在，则说管道已损坏。

然后将其与连接PipedInputStream 哪个实例进入 InputStreamDataSource 然后传入 DataHandler 并返回 DataHandler 的实例。这样你的文件将以块的形式写入，你不会得到那个例外，更多 - 客户端永远不会得到超时。

希望这会有所帮助。

INFO - Sample code

I've set up sample code (SSCCE) for you to help track the problem:

https://github.com/ljader/test-cxf-base64-marshall

The problem

I'm integrating with 3rd party JAX-WS service, so I cannot change the WSDL.

The 3rd party webservice expects Base64 encoded bytes to perform some operation on them - they expect that client sends whole bytes in SOAP message. They don't want to change to MTOM / XOP, so I'm stuck with current requirements.

I decided to use CXF to easily set up sample client, and it worked ok for small files.

But when I try to send BIG data, i.e. 200MB, the CXF/JAXB throws an exception:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.xml.bind.v2.util.ByteArrayOutputStreamEx.readFrom(ByteArrayOutputStreamEx.java:75)
at com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data.get(Base64Data.java:196)
at com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data.writeTo(Base64Data.java:312)
at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.text(UTF8XmlOutput.java:312)
at com.sun.xml.bind.v2.runtime.XMLSerializer.leafElement(XMLSerializer.java:356)
at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$PcdataImpl.writeLeafElement(RuntimeBuiltinLeafInfoImpl.java:191)
at com.sun.xml.bind.v2.runtime.MimeTypedTransducer.writeLeafElement(MimeTypedTransducer.java:96)
at com.sun.xml.bind.v2.runtime.reflect.TransducedAccessor$CompositeTransducedAccessorImpl.writeLeafElement(TransducedAccessor.java:254)
at com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty.serializeBody(SingleElementLeafProperty.java:130)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:360)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:696)
at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl$1.serializeBody(ElementBeanInfoImpl.java:155)
at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl$1.serializeBody(ElementBeanInfoImpl.java:130)
at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeBody(ElementBeanInfoImpl.java:332)
at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeRoot(ElementBeanInfoImpl.java:339)
at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeRoot(ElementBeanInfoImpl.java:75)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:494)
at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:323)
at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:251)
at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.writeObject(JAXBEncoderDecoder.java:617)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:241)
at org.apache.cxf.jaxb.io.DataWriterImpl.write(DataWriterImpl.java:237)
at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:117)
at org.apache.cxf.wsdl.interceptors.BareOutInterceptor.handleMessage(BareOutInterceptor.java:68)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:514)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:423)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:324)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:277)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:139)

My findings

I've tracked bug, that based on xsd type "base64Binary", the

com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl

decides, that

com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data

should handle marshalling of data from

javax.activation.DataHandler

During marshalling, the WHOLE data from underlying InputStream is trying to be read http://grepcode.com/file/repo1.maven.org/maven2/com.sun.xml.bind/jaxb-impl/2.2.11/com/sun/xml/bind/v2/runtime/unmarshaller/Base64Data.java/#311, which causes OOME exception.

Problem

CXF uses JAXB during marshalling Java objects into SOAP messages - when marshalling InputStream, the WHOLE input stream is read to memory before beeing converted into Base64 binary.

So I want to send ("stream") data from client to server in chunks (since the OutputSteam in marshaller is wrapped direct HttpURLConnection), so my client could can handle sending any amount of data.

Especially when many threads would be using my client, the streaming is IMHO very desirable.

I don't have good JAX-WS/CXF/JAXB knowledge, hence the question.

The only materials which I found and may be usefull are:

Can JAXB parse large XML files in chunks

http://rezarahim.blogspot.com/2010/05/chunking-out-big-xml-with-stax-and-jaxb.html

The questions

1. Why CXF/JAXB loads whole InputStream into memory - is not the DataHandler purpouse to prevent such implementation?

2. Do you know any way to change JAXB behaviour to differently marshall InputStream?

3. Do you know different marshallers, which can handle such big data marshalling?

4. As a last resort, maybe you have links to some materials, how to create custom marshaller which would stream the data directly to the server?

解决方案

You don't need any custom marshallers or change JAXB behaviour to achieve what you need - DataHandler is your friend here.

Answering your first question: JAXB needs to keep all data in memory because it has to resolve references.

I know you can't change the WSDL references, etc. But still you do have your client's WSDL in your project in order to generate client classes, don't you? So what you can do (I haven't tested this with third party's WSDL but might be worth trying) is to add xmime:expectedContentTypes="application/octet-stream" into the response XSD element which returns Base64 encoded data. For e.g.:

<xsd:element name="generateBigDataResponse">
    <xsd:complexType>
        <xsd:sequence>
            <xsd:element name="result"
                         type="xsd:base64Binary"
                         minOccurs="0"
                         maxOccurs="1"
                         xmime:expectedContentTypes="application/octet-stream"/>
        </xsd:sequence>
    </xsd:complexType>
</xsd:element>

Also do not forget to add namespace: xmlns:xmime="http://www.w3.org/2005/05/xmlmime" in the xsd:schema element.

What you are doing here - is not changing any WSDL references, just telling JAXB instead of generating byte[] to generate DataHandler. So what happens when you generate your client classes like that:

@Override
public DataHandler generateBigData() {
    try {
        final PipedOutputStream pipedOutputStream = new PipedOutputStream();
        PipedInputStream pipedInputStream = new PipedInputStream(pipedOutputStream);
        InputStreamDataSource dataSource = new InputStreamDataSource(pipedInputStream, "application/octet-stream");

        executor.execute(new Runnable() {

            @Override
            public void run() {
                //write your stuff here into pipedOutputStream
            }
        });

        return new DataHandler(dataSource);
    } catch (IOException e) {
        //handle exception if any
    }
}

You get DataHandler as a response type thanks to xmime. I suggest you use PipedOutputStream, but make sure do the writing in a different thread:

A piped output stream can be connected to a piped input stream to create a communications pipe. The piped output stream is the sending end of the pipe. Typically, data is written to a PipedOutputStream object by one thread and data is read from the connected PipedInputStream by some other thread. Attempting to use both objects from a single thread is not recommended as it may deadlock the thread. The pipe is said to be broken if a thread that was reading data bytes from the connected piped input stream is no longer alive.

Then you connecting it with the PipedInputStream which instance goes into constructor of InputStreamDataSource which you then pass into DataHandler and return DataHandler's instance. This way your file will be written in chunks and you won't get that exception, more - client will never get the timeout.

Hope this helps.

这篇关于为什么CXF / JAXB在编组到SOAP消息之前将整个InputStream读入内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！