SQLite还是平面文本文件？ [英] SQLite or flat text file?

问题描述

我处理了很多在Python，R和有时是Matlab之间交换的文本/数据。

我的go-to是平面文本文件，使用SQLite偶尔存储数据和访问从每个程序（不是Matlab虽然）。我在SQL中不使用GROUPBY，AVG等，我在R中使用这些操作，所以我不一定需要数据库操作。

这样的应用程序需要在程序之间交换数据以利用每种语言的可用库，是否有一个很好的经验法则：使用哪种数据交换格式/方法（甚至XML或NetCDF或HDF5）？

我知道Python之间 - > R有rpy或rpy2，但我想知道这个问题在一个更一般的意义 - 我使用许多计算机，它们都没有rpy2，还使用一些其他

解决方案

如果所有的语言都支持SQLite - 使用它。 SQL的强大功能现在可能对你没有用，但它可能会在某个时候，它可以节省你不得不重写的东西，当你决定你希望能够以更复杂的方式查询你的数据。 p>

如果您只想访问数据存储区中的某些位数据，SQLite也可能会快得多 - 因为使用平面文本文件这样做是有挑战性的，没有读取整个文件（虽然不是不可能）。

I process a lot of text/data that I exchange between Python, R, and sometimes Matlab.

My go-to is the flat text file, but also use SQLite occasionally to store the data and access from each program (not Matlab yet though). I don't use GROUPBY, AVG, etc. in SQL as much as I do these operations in R, so I don't necessarily require the database operations.

For such applications that requires exchanging data among programs to make use of available libraries in each language, is there a good rule of thumb on which data exchange format/method to use (even XML or NetCDF or HDF5)?

I know between Python -> R there is rpy or rpy2 but I was wondering about this question in a more general sense - I use many computers which all don't have rpy2 and also use a few other pieces of scientific analysis software that require access to the data at various times (the stages of processing and analysis are also separated).

解决方案

If all the languages support SQLite - use it. The power of SQL might not be useful to you right now, but it probably will be at some point, and it saves you having to rewrite things later when you decide you want to be able to query your data in more complicated ways.

SQLite will also probably be substantially faster if you only want to access certain bits of data in your datastore - since doing that with a flat-text file is challenging without reading the whole file in (though it's not impossible).

这篇关于SQLite还是平面文本文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！