SQLite3和多处理 [英] SQLite3 and Multiprocessing

问题描述

我注意到，在多处理环境中使用sqlite3时，它并不是真正的功能，也不可靠.每个进程尝试将一些数据写入同一数据库，以便多个线程使用一个连接.我使用check_same_thread = False选项进行了尝试，但是插入次数是非常随机的:有时它包含所有内容，有时不包含所有内容.我应该只并行处理函数的一部分(从Web上获取数据)，将其输出堆叠到一个列表中，然后将它们全部放入表中，还是有一种可靠的方法来使用sqlite处理多连接?

I noticed that sqlite3 isn´t really capable nor reliable when i use it inside a multiprocessing enviroment. Each process tries to write some data into the same database, so that a connection is used by multiple threads. I tried it with the check_same_thread=False option, but the number of insertions is pretty random: Sometimes it includes everything, sometimes not. Should I parallel-process only parts of the function (fetching data from the web), stack their outputs into a list and put them into the table all together or is there a reliable way to handle multi-connections with sqlite?

推荐答案

sqlitedict :围绕Python的sqlite3数据库的轻量级包装，具有类似dict的界面和多-线程访问支持.

sqlitedict: A lightweight wrapper around Python's sqlite3 database, with a dict-like interface and multi-thread access support.

这篇关于SQLite3和多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！