避免并发编程中的死锁 [英] Avoiding deadlocks in concurrent programming

问题描述

这不是特定于Python的，但我知道Python程序员是世界上最好的程序员之一。我对所涉及的

概念有一个公平的理解，足以意识到我将从其他人的

体验中受益:)


我在内存中有一系列共享对象，可能是> 100MB。通常

为客户执行任务必须使用其中几个对象。

因为许多客户可以一次发出请求（>每秒100次峰值加载时
）这些对象必须用某种方法保护

线程同步（threading.Lock会很好。）这有点

复杂的一个备份线程，每10分钟获取一次数据

并将其全部导出为可以保存到磁盘的格式。显然，

备份线程必须对

一次具有对所有这些对象的独占访问权限，并且必须没有半完成的事务正在进行中。

从设计角度来看，最简单的方法就是拥有一个锁，让

的客户可以独家访问所有内容，即使他们只需要
就需要访问一小部分数据。这使得

备份线程变得简单，并确保在部分完成的事务中死锁或

没有问题。但是想象一下，如果100个线程中的100个客户端等待访问那个

锁定，那么会发生什么？一个人几乎可以用它来完成，然后100个线程可以进入和关闭
，所有都无所事事，因为他们都在等待

for the lock持有一个帖子。


所以我认为你需要多个锁，所以客户只需要获得他们需要的
。这将允许多个线程一次访问数据。但是现在我必须处理死锁，因为客户通常会获得一个

资源，然后阻止获取另一个。很可能一个客户锁定了A，另一个锁定了B，然后B的家伙等待A和

A等待B的人。更糟糕的是备份线程将绕过

试图锁定所有内容并且无疑会使每个人陷入僵局。


你如何避免这种情况？我想，而不是永远阻止你可以

尝试在一个超时几秒的循环中获得非阻塞，如果

达到超时，释放任何锁定并稍后再试，

备份线程是唯一一个使用阻塞的人获得

，因为它永远不会完成它的任务。


毫无疑问这是一个常见的问题，你们会怎样处理它？<​​br />

-Dan

解决方案

" Eloff" < EL ****** @ yahoo.com>写道：

我在内存中有一个共享的对象系列，可能是> 100MB。通常，要为客户端执行任务，必须使用其中的几个对象。


你的意思是一些20欧元以上的记录，或者数百万的b / b
几十字节的记录，还是什么？

但是想象一下如果100个
线程中的100个客户端等待访问该锁定会发生什么。一个人几乎可以用它完成，然后100个线程可以进入和关闭，所有这些都没有，因为他们都在等待一个线程所持有的锁定。


如果100个线程被阻塞等待锁定，他们就不应该在锁定被释放之前被唤醒
。因此，如果您可以最大限度地缩短每笔交易的锁定时间，这种方法是合理的。

毫无疑问，这是一个常见问题，您会如何处理它？<​​/ blockquote>


你正在做每个严肃的数据库实现需要做的事情，

和整本书都是关于方法的。一种方法是使用像数据库那样的事务和回滚系统来实现
。

很大程度上取决于你正在做什么的细节。你确定

你不想只使用RDBMS吗？

你好Paul，

你的意思是每个20多MB的记录，或几十个字节的数百万条记录，或者是什么？


好​​吧，他们是列表，词典和数据成员的对象，以及他们内部的其他对象。有些非常大，可能比
20MB更大，而其他的则非常多而且很小（大型列表中有十万个小b / b
）。一些大型物体内部可能有许多

锁，以允许同时访问不同的部件，

而其他部件则需要作为一个单元进行访问。

如果100个线程被阻塞等待锁定，它们就不会被唤醒直到锁定被释放。因此，如果您可以最小化每个事务的锁定时间，那么这种方法是合理的。


现在这很有意思，因为如果100个客户端必须在一秒内完成

系统，服务器显然能够发送100个/ br / >
客户在一秒钟内完成，如果他们都通过立刻进入
并不重要。或者只要一个人没有卡住等待

的时间超过几秒钟。这将是非常简单的，并且我可以毫不费力地将它们一次一个地发送出去。它也可能是某些对象永远不会在同一个动作中被访问，

和那些可能有单独的锁作为优化（这将是

需要仔细分析不同的行为。）

你正在做每个严肃的数据库实现需要做的事情......
你确定你不想只是使用一个RDBMS？

有人考虑过了，但是我们决定将数据抽象到表中，用SQL查询操作
要复杂得多对于

程序员和系统来说太昂贵，因为平均

动作需要20-100个查询。


谢谢，

-Dan

Eloff写道：

嗨Paul，

如果100个线程被阻塞等待锁定，它们就不会被唤醒，直到锁定被释放。因此，如果您可以最大限度地缩短每笔交易的锁定时间，这种方法是合理的。

现在这很有趣，因为如果有100个客户需要通过
系统在一秒钟内，服务器显然能够在一秒钟内发送100个客户端，并且它们是否全部通过立刻进入是无关紧要的。只要没有人等待超过几秒钟，就可以一次一个。对我来说，一次一个地发送它们将是非常简单和无痛的。还有可能某些对象永远不会在同一个动作中被访问，并且那些可能具有单独的锁作为优化（这将需要仔细分析不同的动作。）

据我所知，Pythons多线程是在

interpteter级别完成的，并且解释器本身是单一的

螺纹。在这种情况下，即使在多CPU机器上也不能让多个线程同时运行

，所以只要你拿着锁就可以避免I / O工作，我认为不应该使用单一锁来获得任何性能。备份线程可能会是一个问题。


Steve

This is not really Python specific, but I know Python programmers are
among the best in the world. I have a fair understanding of the
concepts involved, enough to realize that I would benefit from the
experience of others :)

I have a shared series of objects in memory that may be > 100MB. Often
to perform a task for a client several of these objects must be used.
Since many clients can be making requests at once (>100 per second
during peak load) these objects must be protected with some method of
thread syncronization (threading.Lock would do fine.) This is somewhat
complicated by a backup thread which takes the data every 10 minutes
and exports it all to a format that can be saved to disk. Clearly the
backup thread must have exclusive access to all of these objects at
once, and there must be no half-completed transactions underway.

The easiest way from a design stand point is have a single lock and let
clients have exclusive access to everything even although they only
ever need access to a fraction of the data. This makes it easy for the
backup thread, and it ensures that there''s no trouble with deadlocks or
with partially completed transactions. However imagine what would
happen if there''s 100 clients in 100 threads waiting for access to that
lock. One could be almost finished with it, and then 100 threads could
get switched in and out, all doing nothing since they''re all waiting
for the lock held by the one thread.

So I think you would need multiple locks so clients only acquire what
they need. This would let multiple threads access the data at once. But
now I have to deal with deadlocks since clients will usually acquire a
resource and then block acquiring another. It is very likely that one
client locks A, another locks B, then the guy with B waits for A and
the guy with A waits for B. Worse yet the backup thread will go around
trying to lock everything and will no doubt deadlock everybody.

How do you avoid this? I thought instead of blocking forever you could
try non-blocking acquires in a loop with a timeout of a few seconds, if
the timeout is reached, release any locks held and try again later,
with the backup thread being the only one to use blocking acquires
since it would never complete it''s task otherwise.

No doubt this is a common problem, how would you people deal with it?

-Dan

解决方案

"Eloff" <el******@yahoo.com> writes:

I have a shared series of objects in memory that may be > 100MB. Often
to perform a task for a client several of these objects must be used.
Do you mean a few records of 20+ MB each, or millions of records of a
few dozen bytes, or what?
However imagine what would happen if there''s 100 clients in 100
threads waiting for access to that lock. One could be almost finished
with it, and then 100 threads could get switched in and out, all doing
nothing since they''re all waiting for the lock held by the one thread.
If the 100 threads are blocked waiting for the lock, they shouldn''t
get awakened until the lock is released. So this approach is
reasonable if you can minimize the lock time for each transaction.
No doubt this is a common problem, how would you people deal with it?

You''re doing what every serious database implementation needs to do,
and whole books have been written about approaches. One approach is
to use a transaction and rollback system like a database does.
A lot depends on the particulars of what you''re doing. Are you sure
you don''t want to just use an RDBMS?

Hi Paul,

Do you mean a few records of 20+ MB each, or millions of records of a
few dozen bytes, or what?
Well they''re objects with lists and dictionaries and data members and
other objects inside of them. Some are very large, maybe bigger than
20MB, while others are very numerous and small (a hundred thousand tiny
objects in a large list). Some of the large objects could have many
locks inside of them to allow simultaneous access to different parts,
while others would need to be accessed as a unit.
If the 100 threads are blocked waiting for the lock, they shouldn''t
get awakened until the lock is released. So this approach is
reasonable if you can minimize the lock time for each transaction.
Now that is interesting, because if 100 clients have to go through the
system in a second, the server clearly is capable of sending 100
clients through in a second, and it doesn''t matter if they all go
through "at once" or one at a time so long as nobody gets stuck waiting
for much longer than a few seconds. It would be very simple and
painless for me to send them all through one at a time. It is also
possible that certain objects are never accessed in the same action,
and those could have seperate locks as an optimization (this would
require carefull analysis of the different actions.)
You''re doing what every serious database implementation needs to do ...
Are you sure you don''t want to just use an RDBMS?

It was considered, but we decided that abstracting the data into tables
to be manipulated with SQL queries is substantially more complex for
the programmer and way too expensive to the system since the average
action would require 20-100 queries.

Thanks,
-Dan

Eloff wrote:

Hi Paul,

If the 100 threads are blocked waiting for the lock, they shouldn''t
get awakened until the lock is released. So this approach is
reasonable if you can minimize the lock time for each transaction.

Now that is interesting, because if 100 clients have to go through the
system in a second, the server clearly is capable of sending 100
clients through in a second, and it doesn''t matter if they all go
through "at once" or one at a time so long as nobody gets stuck waiting
for much longer than a few seconds. It would be very simple and
painless for me to send them all through one at a time. It is also
possible that certain objects are never accessed in the same action,
and those could have seperate locks as an optimization (this would
require carefull analysis of the different actions.)

It is my understanding that Pythons multithreading is done at the
interpteter level and that the interpreter itself is single
threaded. In this case, you cannot have multiple threads running
truly concurrently even on a multi-CPU machine, so as long as you
avoid I/O work while holding the lock, I don''t think there should
be any performance hit using a single lock. The backup thread may
be an issue though.

Steve

这篇关于避免并发编程中的死锁的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！