列表不会随着 Ray 并行 python 改变 [英] Lists won't change with Ray parallel python

问题描述

我的问题是，如果我重新分配列表中的一个项目，使得重新分配发生在并行进程中，那么在并行进程完成后，更改将恢复到其原始状态.

My issue is that if I reassign an item in a list such that the reassignment happens during a parallel process, then after the parallel processes are finished, the change reverts back to its original state.

在下面的示例中 - 为便于理解而进行了大大简化 - 我有一个函数将列表元素 NoZeros[0] 更改为chicken"，第二个函数将 NoZeros[1] 更改为sandwich".我什至在第二个函数中加入了global"，只是为了证明这不是局部与全局的问题——它看起来像一个问题，但实际上不是.当您运行程序时，打印命令向您显示，列表元素确实发生了变化.问题是在这些过程之后调用 NoZeros 时，NoZeros 是它的开头，而不是[chicken", sandwich"].

In the below example- greatly simplified for ease of understanding-, I have a function that changes the list element NoZeros[0] to "chicken" and a second function that changes NoZeros[1] to "sandwich". I even put "global" in the second function just to demonstrate that this isn't a local vs global issue- it looks like one, but it indeed is not. As the print commands show you when you run the program, the list elements do actually change. The issue is that when calling NoZeros after these processes, NoZeros is what it was to begin with instead of "["chicken", "sandwich"].

我知道 python 的 multiprocessing 包有完全相同的问题，但它是通过一步"解决的.我的问题是，我一生都无法弄清楚 Ray 的等价物是什么.例如，在 Python 的多处理库中，您只需在 NoZeros 被更改之前的某处编写 NoZeros=manager.list(NoZeros)，这将是它的结束，但我无法找到 Ray 的等效项，或者如果有even 是等价的.

I know that python's multiprocessing package has exactly the same issue, but it was solved by taking "one step up" of whatever it is you wanted to not revert back to and slapping "manager.list()" at the before it. My issue is that I can't for the life of me figure out what the equivalent is for Ray. For example, in Python's multiprocessing library, you would just write NoZeros=manager.list(NoZeros) somewhere before NoZeros gets altered, and that would be the end of it, but I can't find what equivalent there is for Ray or if there even is an equivalent.

如何使用 RAY 更改并行列表?非常感谢.

HOW DO I CHANGE LISTS IN PARALLEL USING RAY? Many thanks.

另外:请注意，此脚本可能会让您陷入循环，因为您可能会在并行进程完成之前打印 NoZeros.这是我遇到的另一个错误，希望得到关注，但这不是优先事项.我想说明的一点是，您可能希望在下一个单元格中运行 print(NoZeros) (至少 Jupyter 具有此功能).在 python 的多处理库中，一个人只需执行process.join()"就可以解决有问题的进程，所以这让我想到了额外的问题:

Also: note that this script may throw you for a loop, because you may end up printing NoZeros BEFORE the parallel processes finish. This is another bug I'm having trouble with and would appreciate attention on, but it is not the priority. The point I'm trying to make is that you probably want to run print(NoZeros) in the next cell (Jupyter has this functionality, at least). In python's multiprocessing library, one would just do "process.join()" and that would solve end the process(es) in question, so that brings me to the bonus question:

额外问题:我如何让 ray.wait() 工作；如果前面的命令(即使这些是并行命令)已完成，我该如何告诉我的代码仅继续执行下一个命令?

Bonus question: How do i get ray.wait() to work; how do I tell my code to only proceed to my next command if the previous commands- even if these are parallel commands- are finished?

'''

import ray

ray.shutdown()
ray.init()

NoZeros=[0,0]

@ray.remote
def Chicken():
    print("NoZeros[0] is",NoZeros[0],"but will change to chicken")
    NoZeros[0]="chicken"
    print("Now, NoZeros[0] is",NoZeros[0])


@ray.remote
def GlobalSandwich():
    global NoZeros #This is just to show that "global" doesn't solve anything
    print("NoZeros[1] is",NoZeros[1],"but will change to sandwich")
    NoZeros[1]="sandwich"
    print("Now, NoZeros[1] is",NoZeros[1])

Chicken.remote()
GlobalSandwich.remote()

#Unhash these 3 lines of code if youd like to try tackling another question: why does ray.wait() not work? 
#How do i wait until parallel processes end, so i can continue my code?

#WaitList=[Chicken.remote(),GlobalSandwich.remote()]
#ray.wait(WaitList)
#print("If you see this first, ray.wait() isnt working")


#This line of code right here was executed in the next command line (Jupyter); this print command happens when the other processes are finished  
print(NoZeros)

'''

推荐答案

对于 Ray，可变全局状态应该存在于 演员.例如，您可以执行以下操作:

With Ray, mutable global state should live in Actors. For example, you could do something like:

@ray.remote
class ListActor:
    def __init__(self, l):
        self._list = l

    def get(self, i):
        return self._list[i]

    def set(self, i, val):
        self._list[i] = val

    def to_list(self):
        return self._list

然后为了使用它，你可以将它作为参数传入(同样，你不应该依赖全局变量).

Then in order to use it, you can pass it in as a parameter (again, you shouldn't rely on global variables).

NoZeros = ListActor.remote([0,0])

@ray.remote
def Chicken(NoZeros):
    print("NoZeros[0] is",ray.get(NoZeros.get.remote(0)),"but will change to chicken")
    NoZeros.set(0, "chicken")
    print("Now, NoZeros[0] is",ray.get(NoZeros.get(0)))


# We need to make sure this function finishes executing before we print.
ray.get(Chicken.remote(NoZeros))

print(ray.get(NoZeros.to_list.remote()))

这篇关于列表不会随着 Ray 并行 python 改变的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！