python pickle vs sql效率 [英] python pickle vs sql efficiency

问题描述

我正在用 Python 开发一个应用程序，它需要存储(非常)大的数据集.pickle 是存储数据并根据请求检索数据的最实用方法，还是应该考虑改用 SQL?我的主要目标是速度和尽可能少的处理压力.

I'm developing an application in Python which requires storing (very) large data sets. Is pickle the most practical way to store the data and retrieve it on request, or should i consider using SQL instead? My main goals are speed and as little processing strain as possible.

我担心的是，pickle 必须动态处理整个大文件，这可能会对性能产生不利影响.除了使用之外，我对泡菜并不是特别熟悉，因此对其工作原理的任何解释都会很棒.

My concern is that pickle has to process an entire large file on the fly, which could adversely effect performance. I'm not particularly familiar with pickle beyond usage, so any explanation to how it works would be great.

现在，我正在使用此代码:

Right now, i'm using this code:

users = pickle.load( open( "users.py", "rb" ) )
username = raw_input("Please enter a username: ")
password = raw_input("Please enter a password: ")
var = username in users
if(var == 0):
    return 0
    exit()
else:
    if(users[username] != password):
        return 0
        exit()
    else:
        return 1
        exit()

想象用户包含 100 万个条目，这还是 SQL 效率更高?

Imaging that users contains 1 million entries, which would be more efficient, this or SQL?

任何帮助都会很棒，

谢谢

推荐答案

Pickle 通常适用于对象的存储，如果您想高效地存储原始"数据，那么 Pickle 可能不是要走的路，但它非常取决于具体情况 - 加载"数据时间很关键，您是否有开发时间来设置数据库、查询等.

Pickle is generally suited to storage of objects, if you want to store 'raw' data efficiently then pickle probably isn't the way to go, but its very dependant on the specific situation - is 'loading' the data time critical, do you have the development time to set up a database, queries etc.

如果您的数据是一百万对用户名和出生日期，那么 pickle 可能不是最好的方法，可以说将数据存储在纯文本文件中会更简单.

If your data is a million pairs of username and date of birth then pickle is probably not the best way to go, it would be arguably simpler to store the data in a flat text file.

pickle 和 db/SQL 解决方案都具有可扩展的优势.请记住，pickle 并不安全"，因此您应该考虑文件的可信度，例如是否会在不同系统之间转移.

Both the pickle and the db/SQL solutions have the advantage of being extendible. Bear in mind pickle is not 'secure' and so you should consider the trustworthiness of the file, e.g. Would it be transferred between different systems.

总的来说，如果你的数据集非常大，关系数据库可能比pickle更合适，但你可能还想考虑其他存储引擎，例如.Redis、MongoDb、Memcached.不过，所有这些都非常依赖于具体情况，因此您可以提供有关预期如何使用数据的更多信息！

Overall, if your data sets are very large, a relational Db may be more suitable than pickle, but you may also want to consider other storage engines, e.g. Redis, MongoDb, Memcached. All of them, are very situation dependent though, so any more info you can provide on how the data is expected to be used would be useful!

这篇关于python pickle vs sql效率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！