如何使用Hibernate将数据流传输到数据库BLOB（在byte []中没有内存存储） [英] How to stream data to database BLOB using Hibernate (no in-memory storing in byte[])

问题描述

我正在寻找一种方式来流式传输二进制数据到/从数据库。如果可能，我想用Hibernate（在数据库不可知的方式）。
我发现的所有解决方案涉及将二进制数据显式或隐式加载到内存中作为byte []。我需要避免它。假设我想让我的代码能够从本地文件写入一个2GB的视频从数据库（存储在BLOB列），或者反过来，使用不超过256Mb的内存。这显然是可以实现的，不涉及巫毒。但我找不到一个办法，现在我试图避免调试Hibernate。

I'm looking for a way to stream binary data to/from database. If possible, i'd like it to be done with Hibernate (in database agnostic way). All solutions I've found involve explicit or implicit loading of binary data into memory as byte[]. I need to avoid it. Let's say I want my code to be able to write to a local file a 2GB video from database (stored in BLOB column), or the other way around, using no more than 256Mb of memory. It's clearly achievable, and involves no voodoo. But I can't find a way, for now I'm trying to avoid debugging Hibernate.

让我们看看示例代码（记住-Jmx = 256Mb）。

Let's look at sample code (keeping in mind -Jmx=256Mb).

实体类：

public class SimpleBean {
    private Long id;
    private Blob data;
    // ... skipping getters, setters and constructors.
}

Hibernate映射片段：

Hibernate mapping fragment:

<class name="SimpleBean" table="SIMPLE_BEANS">
    <id name="id" column="SIMPLE_BEAN_ID">
        <generator class="increment" />
    </id>
    <property name="data" type="blob" column="DATA" />
</class>

测试代码片段：

Configuration cfg = new Configuration().configure("hibernate.cfg.xml");
ServiceRegistry serviceRegistry = new ServiceRegistryBuilder()
                                      .applySettings(cfg.getProperties())
                                      .buildServiceRegistry();

SessionFactory sessionFactory = cfg.buildSessionFactory(serviceRegistry);
Session session = sessionFactory.openSession();
session.beginTransaction();

File dataFile = new File("movie_1gb.avi");
long dataSize = dataFile.length();
InputStream dataStream = new FileInputStream(dataFile);

LobHelper lobHelper = session.getLobHelper();
Blob dataBlob = lobHelper.createBlob(dataStream, dataSize);

session.save( new SimpleBean(data) );
session.getTransaction().commit(); // Throws java.lang.OutOfMemoryError
session.close();

blobStream.close();
sessionFactory.close();

当运行该代码片段时，我得到OutOfMemory异常。查看堆栈跟踪显示什么Hibernate尝试加载流在内存中并获取OutOfMemory（因为它应该）。这里是堆栈跟踪：

When running that snippet I get OutOfMemory exception. Looking at stack trace shows what Hibernate tries to load the stream in memory and gets OutOfMemory (as it should). Here's stack trace:

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at org.hibernate.type.descriptor.java.DataHelper.extractBytes(DataHelper.java:183)
at org.hibernate.type.descriptor.java.BlobTypeDescriptor.unwrap(BlobTypeDescriptor.java:121)
at org.hibernate.type.descriptor.java.BlobTypeDescriptor.unwrap(BlobTypeDescriptor.java:45)
at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$4$1.doBind(BlobTypeDescriptor.java:105)
at org.hibernate.type.descriptor.sql.BasicBinder.bind(BasicBinder.java:92)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:305)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:300)
at org.hibernate.type.AbstractSingleColumnStandardBasicType.nullSafeSet(AbstractSingleColumnStandardBasicType.java:57)
at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:2603)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2857)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3301)
at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:88)
at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:362)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:354)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:275)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:326)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1214)
at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:403)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:175)
at ru.swemel.msgcenter.domain.SimpleBeanTest.testBasicUsage(SimpleBeanTest.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)

使用Hibernate 4.1.5.SP1。确切的问题是：如何避免加载流到内存当使用Hibernate在数据库中存储blob时，使用直接流。我想避免关于为什么一个存储视频在数据库列，而不是存储在一些内容存储库和链接的主题。请考虑它是一个与这个问题无关的模型。

Used Hibernate 4.1.5.SP1. The exact question is: how to avoid loading stream into memory when storing a blob in database using Hibernate, using direct streaming instead. I'd like to avoid off topics about why one stores video in column of database instead of storing it in some content repository and linking. Please, consider it a model what is irrelevant to the question.

似乎在不同的方言可能有某种功能，Hibernate可能会尝试加载内存中的一切，因为底层数据库不支持流blob或类似的东西。如果是这样 - 我想在处理Blob的方面看到不同方言之间的某种比较表。

It seems that there might be some kind of capabilities on different dialects and Hibernate might try to load everything in memory, because underlying database doesn't support streaming blobs or something like that. If it's the case - i'd like to see some kind of comparative table between different dialects in aspect of handling blobs.

非常感谢您的帮助！ p>

Thank you very much for your help!

推荐答案

对于那些寻找同样东西的人。

For those looking for same thing.

工作原理（流不试图复制到内存）PostgreSQL（和可能很多其他人）。 Hibernate的内部工作取决于所选择的方言。我用的第一个覆盖直接使用流，有利于由byte []支持的BinaryStream。

My bad, the code works as supposed to (streams without trying to copy to memory) for PostgreSQL (and probably lots of others). The inner work of Hibernate depends on selected dialect. The one I used in the first place overrides direct use of streams in favor of BinaryStream backed by byte[].

也没有性能问题，因为它只加载OID（number）在PostgreSQL的情况下，并且可能延迟加载数据在其他方言（包括byte []实现）。只是运行了一些脏的测试，在有和没有二进制数据字段的实体的10000加载没有可见差异。

Also there are no problems with performance, since it loads only OID (number) in case of PostgreSQL, and probably lazy loads data in case of other dialects (including byte[] implementation). Just ran some dirty tests, no visible difference in 10000 loads of entity with and without binary data field.

在数据库中存储数据似乎比只是保存在磁盘作为外部文件。但是，当备份或处理特定文件系统的限制或并发更新等时，它可以节省很多头痛。但是这是一个偏离主题。

Storing data in database seems to be slower than just saving it on disk as external file though. But it saves you a lot of headache when backing up, or dealing with limitations of particular file system, or concurrent updates, etc. But it's an off-topic.

这篇关于如何使用Hibernate将数据流传输到数据库BLOB（在byte []中没有内存存储）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！