python中的多线程应用中的分段错误错误 [英] Segmentation fault error in a multi threaded app in python

问题描述

我在python中有一个多线程应用程序，其中我创建了多个生产者线程，它们从数据库中提取数据.数据是按块提取的.因此，线程创建具有限制值的sql语句的部分保持锁定状态.为了让线程同时执行查询，query()函数将保留在锁之外.然后将结果获取部分再次保留在锁中.下面是代码片段:

I have a multi threaded app in python, wherein I create multiple producer threads and they extract the data from DB. Data is extracted in chunks. So the part where a thread creates sql statement with limit values is kept within lock. And to let threads execute queries simultaneously, query() function is kept outside the lock. Then the result fetching part is again kept under the lock. Below is the code snippet:

with UserAgent.lock:
    sqlGeoTarget = "call sp_ax_ari_select_user_agent_list('0'," + str(self.chunkStart) + "," + str(self.chunkSize) + ",1);"
    self.chunkStart += self.chunkSize

self.dbObj.query(sqlGeoTarget)
print "query executed. Processing data now..."+sqlGeoTarget

with UserAgent.lock:
    result = self.dbObj.fetchAll()
    self.dbObj.dbCursor.close()

但是此代码生成致命错误segmentation fault (core dumped).因为如果我将所有代码置于锁定状态，它会很好地执行.我在获取数据后显式关闭了游标，当再次触发query()函数时，游标被重新打开.

But this code generates fatal error segmentation fault (core dumped). Because if I put all the code under lock, it executes fine. I explicitly close the cursor after fetching the data, it is reopened when query() function fired again.

此代码位于名为UserAgent的类中，并且是名为Producer的类的共享资源.因此，数据库对象是共享的.因此，问题区域99％必须是因为db对象是共享的，所以同时击中了查询并关闭了游标，那么必须弄乱了结果集.但是，如何解决这个问题并实现并发的数据库查询执行呢?

This code is inside a class named UserAgent and it's a shared resource for a class named Producer. Thus, database object is shared. So the problem area 99% must be that as the db object is shared hitting query simultaneously and closing cursor then must be messing up with result set. But then how to solve this problem and achieve concurrent db query execution?

推荐答案

请勿在线程之间重用连接.而是为每个线程创建一个新的连接.

Do not reuse connections across threads. Create a new connection for each thread instead.

来自《 MySQLdb用户指南》:

From the MySQLdb User Guide:

MySQL协议不能一次使用同一连接处理多个线程. MySQLdb的某些早期版本利用锁定来实现2的线程安全.尽管使用标准Cursor类(使用mysql_store_result())并不是很难做到这一点，但是SSCursor(使用mysql_use_result())使它变得复杂.您必须确保已读取所有行，然后才能执行另一个查询，这会因添加事务而变得更加复杂，因为事务在游标执行查询时开始，而在COMMIT或ROLLBACK被执行时结束.连接对象:除了在查询执行期间无法共享连接外，两个线程根本无法在事务进行过程中共享连接.这使代码过于复杂，以至于它根本不值得.

The MySQL protocol can not handle multiple threads using the same connection at once. Some earlier versions of MySQLdb utilized locking to achieve a threadsafety of 2. While this is not terribly hard to accomplish using the standard Cursor class (which uses mysql_store_result()), it is complicated by SSCursor (which uses mysql_use_result(); with the latter you must ensure all the rows have been read before another query can be executed. It is further complicated by the addition of transactions, since transactions start when a cursor execute a query, but end when COMMIT or ROLLBACK is executed by the Connection object. Two threads simply cannot share a connection while a transaction is in progress, in addition to not being able to share it during query execution. This excessively complicated the code to the point where it just isn't worth it.

通常的结果是:不要在线程之间共享连接.这确实不值得您费力，最后，可能会损害性能，因为MySQL服务器为每个连接运行一个单独的线程.您当然可以做一些事情，例如在池中缓存连接，然后一次将这些连接分配给一个线程.如果让两个线程同时使用连接，则MySQL客户端库可能会崩溃.您已被警告.

The general upshot of this is: Don't share connections between threads. It's really not worth your effort or mine, and in the end, will probably hurt performance, since the MySQL server runs a separate thread for each connection. You can certainly do things like cache connections in a pool, and give those connections to one thread at a time. If you let two threads use a connection simultaneously, the MySQL client library will probably upchuck and die. You have been warned.

强调我的.

使用线程本地存储或专用连接池库.

这篇关于python中的多线程应用中的分段错误错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！