关于NetCDF和HDF5用于存储科学数据的观点? [英] Opinions on NetCDF vs HDF5 for storing scientific data?

问题描述

任何人都有NetCDF和HDF5的丰富经验，可以为它们提供一些优点/缺点，作为存储科学数据的一种方式?

Anyone out there have enough experience w/ NetCDF and HDF5 to give some pluses / minuses about them as a way of storing scientific data?

我使用过HDF5，并且想通过Java进行读写，但是接口本质上是C库的包装，这让我感到困惑，因此NetCDF似乎很吸引人，但我对此一无所知.

I've used HDF5 and would like to read/write via Java but the interface is essentially a wrapper around the C libraries, which I have found confusing, so NetCDF seems intriguing but I know almost nothing about it.

编辑:我的应用程序仅用于数据记录，因此我得到的文件具有自描述格式.对我来说重要的功能是能够添加任意元数据，具有快速写入访问权限以附加到字节数组以及具有单写入器/多读取器并发性(强烈建议但并非必须并发.NetCDF文档说它们具有SWMR，但没有不会说他们是否支持任何机制来确保两个编写者无法一次打开同一个文件而导致灾难性的结果).我喜欢HDF5的层次结构方面(特别是我爱有向无环图层次结构，比像常规"文件系统一样的层次结构更加灵活)，现在正在阅读NetCDF文档...如果每个文件只允许一个数据集，那么它可能对我不起作用. :(

edit: my application is "only" for datalogging, so that I get a file that has a self-describing format. Important features for me are being able to add arbitrary metadata, having fast write access for appending to byte arrays, and having single-writer / multiple-reader concurrency (strongly preferred but not a must-have. NetCDF docs say they have SWMR but don't say whether they support any mechanism for ensuring that two writers can't open the same file at once with disastrous results). I like the hierarchical aspect of HDF5 (in particular I love the directed-acyclic-graph hierarchy, much more flexible than a "regular" filesystem-like hierarchy), am reading the NetCDF docs now... if it only allows one dataset per file then it probably won't work for me. :(

更新—看起来 NetCDF-Java 从netCDF-4文件读取，但仅从netCDF写入-3不支持分层组的文件.该死的.

update — looks like NetCDF-Java reads from netCDF-4 files but only writes from netCDF-3 files which don't support hierarchical groups. darn.

更新2009年7月14日:我开始对Java中的HDF5感到非常沮丧.可用的库不是很好，它有一些主要的绊脚石，它们与Java的抽象层(复合数据类型)有关. C的一种很好的文件格式，但看起来好像我输了. >:(

update 2009-Jul-14: I am starting to get really upset with HDF5 in Java. The library available isn't that great and it has some major stumbling blocks that have to do with Java's abstraction layers (compound data types). A great file format for C but looks like I just lose. >:(

推荐答案

我强烈建议您使用HDF5，而不要使用NetCDF. NetCDF是扁平的，如果您无法对内容进行分类，则它会在一段时间后变得很脏.当然，分类也是一个辩论的问题，但是至少您具有这种灵活性.

I strongly suggest you HDF5 instead of NetCDF. NetCDF is flat, and it gets very dirty after a while if you are not able to classify stuff. Of course classification is also a matter of debate, but at least you have this flexibility.

当我编写Q5Cost时，我们对HDF5与NetCDF进行了准确的评估，最终结果是HDF5放下了手.

We performed an accurate evaluation of HDF5 vs. NetCDF when I wrote Q5Cost, and the final result was for HDF5 hands down.

这篇关于关于NetCDF和HDF5用于存储科学数据的观点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！