如何与子进程共享父进程的numpy随机状态? [英] How to share numpy random state of a parent process with child processes?

问题描述

我在程序开始时设置了numpy随机种子.在程序执行期间，我使用multiprocessing.Process多次运行一个函数.该函数使用numpy随机函数绘制随机数.问题是Process获取当前环境的副本.因此，每个进程都独立运行，并且都以与父环境相同的随机种子开始.

I set numpy random seed at the beginning of my program. During the program execution I run a function multiple times using multiprocessing.Process. The function uses numpy random functions to draw random numbers. The problem is that Process gets a copy of the current environment. Therefore, each process is running independently and they all start with the same random seed as the parent environment.

所以我的问题是如何在父环境中与子进程环境共享numpy的随机状态?请注意，我要在工作中使用Process，并且需要使用单独的类，并在该类中单独进行import numpy.我尝试使用multiprocessing.Manager共享随机状态，但是似乎事情没有按预期进行，并且我总是得到相同的结果.同样，是否将for循环移到drawNumpySamples内还是将其留在main.py中也没有关系.我仍然无法获得不同的数字，并且随机状态始终相同.这是我的代码的简化版本:

So my question is how can I share the random state of numpy in the parent environment with the child process environment? Just note that I want to use Process for my work and need to use a separate class and do import numpy in that class separately. I tried using multiprocessing.Manager to share the random state but it seems that things do not work as expected and I always get the same results. Also, it does not matter if I move the for loop inside drawNumpySamples or leave it in main.py; I still cannot get different numbers and the random state is always the same. Here's a simplified version of my code:

# randomClass.py
import numpy as np
class myClass(self):
    def __init__(self, randomSt):
        print ('setup the object')
        np.random.set_state(randomSt)
    def drawNumpySamples(self, idx)
        np.random.uniform()

在主文件中:

    # main.py
    import numpy as np
    from multiprocessing import Process, Manager
    from randomClass import myClass

    np.random.seed(1) # set random seed
    mng = Manager()
    randomState = mng.list(np.random.get_state())
    myC = myClass(randomSt = randomState)

    for i in range(10):
        myC.drawNumpySamples() # this will always return the same results

注意:我使用的是Python 3.5.我还在Numpy的GitHub页面上发布了一个问题.只需在此处发送问题链接，以备将来参考.

Note: I use Python 3.5. I also posted an issue on Numpy's GitHub page. Just sending the issue link here for future reference.

推荐答案

即使您设法使它正常运行，我也不认为它会做您想要的事情.一旦有多个流程并行地从同一个随机状态中拉出，就不再确定它们分别进入状态的顺序，这意味着您的运行实际上不会重复.可能有一些解决方法，但这似乎是一个不小的问题.

Even if you manage to get this working, I don’t think it will do what you want. As soon as you have multiple processes pulling from the same random state in parallel, it’s no longer deterministic which order they each get to the state, meaning your runs won’t actually be repeatable. There are probably ways around that, but it seems like a nontrivial problem.

同时，有一种解决方案应同时解决您想要的问题和不确定性问题:

Meanwhile, there is a solution that should solve both the problem you want and the nondeterminism problem:

在生成子进程之前，请向RNG索取随机数，然后将其传递给子进程.然后，孩子可以使用该数字进行播种.这样，每个孩子的随机序列都将不同于其他孩子，但是如果您使用固定种子重新运行整个应用程序，则同一个孩子会获得相同的随机序列.

Before spawning a child process, ask the RNG for a random number, and pass it to the child. The child can then seed with that number. Each child will then have a different random sequence from other children, but the same random sequence that the same child got if you rerun the entire app with a fixed seed.

如果您的主流程执行了可能不确定地依赖于子流程执行的任何其他RNG工作，则您需要按顺序为所有子流程预先生成种子，然后再提取其他随机变量数字.

If your main process does any other RNG work that could depend non-deterministically on the execution of the children, you'll need to pre-generate the seeds for all of your child processes, in order, before pulling any other random numbers.

正如senderle在评论中指出的那样:如果您不需要多次不同的运行，而只需进行一次固定运行，则您甚至根本不需要从已播种的RNG中获取种子.只需使用一个从1开始的计数器，并为每个新进程将其递增，并将其用作种子即可.我不知道这是否可以接受，但是如果可以接受，很难比这更简单了.

As senderle pointed out in a comment: If you don't need multiple distinct runs, but just one fixed run, you don't even really need to pull a seed from your seeded RNG; just use a counter starting at 1 and increment it for each new process, and use that as a seed. I don't know if that's acceptable, but if it is, it's hard to get simpler than that.

正如Amir在评论中指出的那样:更好的方法是每次生成新进程时绘制一个随机整数，并将该随机整数传递给新进程，以使用该整数设置numpy的随机种子.该整数确实可以来自np.random.randint().

As Amir pointed out in a comment: a better way is to draw a random integer every time you spawn a new process and pass that random integer to the new process to set the numpy's random seed with that integer. This integer can indeed come from np.random.randint().

这篇关于如何与子进程共享父进程的numpy随机状态?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！