如何在python中跨脚本共享变量? [英] How to share variables across scripts in python?

问题描述

以下不起作用

一个.py

import shared
shared.value = 'Hello'
raw_input('A cheap way to keep process alive..')

两个.py

import shared
print shared.value

在两个命令行上运行:

>>python one.py
>>python two.py

(第二个得到属性错误，这是正确的).

(the second one gets an attribute error, rightly so).

有没有办法做到这一点，即在两个脚本之间共享一个变量?

Is there a way to accomplish this, that is, share a variable between two scripts?

推荐答案

希望可以在这里记下我关于这个问题的笔记.

Hope it's OK to jot down my notes about this issue here.

首先，我非常欣赏 OP 中的示例，因为那也是我开始的地方——尽管它让我认为 shared 是一些内置的 Python 模块，直到我发现[Tutor] 模块之间的全局变量 ?? 上的完整示例.

First of all, I appreciate the example in the OP a lot, because that is where I started as well - although it made me think shared is some built-in Python module, until I found a complete example at [Tutor] Global Variables between Modules ??.

但是，当我寻找在脚本之间共享变量"(或进程)时——除了 Python 脚本需要使用在其他 Python 源文件中定义的变量(但不一定是运行进程)的情况之外——我主要是偶然发现了两个其他用例:

However, when I looked for "sharing variables between scripts" (or processes) - besides the case when a Python script needs to use variables defined in other Python source files (but not necessarily running processes) - I mostly stumbled upon two other use cases:

• 一个脚本将自己分叉成多个子进程，然后在同一台 PC 上并行运行(可能在多个处理器上)
• 一个脚本产生多个其他子进程，然后在同一台 PC 上并行运行(可能在多个处理器上)

因此，大多数关于共享变量"和进程间通信"(IPC)的点击都讨论了这两种情况；然而，在这两种情况下，都可以观察到父母"，孩子"通常有一个参考.

As such, most hits regarding "shared variables" and "interprocess communication" (IPC) discuss cases like these two; however, in both of these cases one can observe a "parent", to which the "children" usually have a reference.

然而，我感兴趣的是运行同一脚本的多个调用，独立运行，并在它们之间共享数据(如 Python:如何在单例/单实例模式下跨多个脚本调用共享对象实例).上述两种情况并没有真正解决这种问题 - 相反，它本质上简化为 OP 中的示例(跨两个脚本共享变量).

What I am interested in, however, is running multiple invocations of the same script, ran independently, and sharing data between those (as in Python: how to share an object instance across multiple invocations of a script), in a singleton/single instance mode. That kind of problem is not really addressed by the above two cases - instead, it essentially reduces to the example in OP (sharing variables across two scripts).

现在，在 Perl 中处理这个问题时，有 IPC::Shareable;它允许您将变量绑定到共享内存"，使用一个整数或 4 个字符的字符串 [1] 作为跨进程空间的数据的通用标识符".因此，没有临时文件，也没有网络设置——我觉得这对我的用例非常有用；所以我在 Python 中寻找相同的东西.

Now, when dealing with this problem in Perl, there is IPC::Shareable; which "allows you to tie a variable to shared memory", using "an integer number or 4 character string[1] that serves as a common identifier for data across process space". Thus, there are no temporary files, nor networking setups - which I find great for my use case; so I was looking for the same in Python.

但是，正如 @Drewfer 接受的答案 指出的那样:您将无法做你想做的事，而无需将信息存储在解释器的两个实例之外的某个地方";或者换句话说:要么您必须使用网络/套接字设置 - 要么您必须使用临时文件(因此，完全独立的 python 会话"没有共享 RAM).

However, as accepted answer by @Drewfer notes: "You're not going to be able to do what you want without storing the information somewhere external to the two instances of the interpreter"; or in other words: either you have to use a networking/socket setup - or you have to use temporary files (ergo, no shared RAM for "totally separate python sessions").

现在，即使有这些考虑，也很难找到工作示例(pickle 除外)——也在 mmap 和 多处理.我设法找到了一些其他示例 - 这些示例也描述了文档未提及的一些陷阱:

Now, even with these considerations, it is kinda difficult to find working examples (except for pickle) - also in the docs for mmap and multiprocessing. I have managed to find some other examples - which also describe some pitfalls that the docs do not mention:

• mmap 的使用:使用 mmap 在进程之间共享 Python 数据 |施迈克尔的博客
  • 演示两个脚本如何更改共享值
  • 注意这里创建了一个临时文件作为保存数据的存储空间 - mmap 只是访问这个临时文件的一个特殊接口
  • Usage of mmap: working code in two different scripts at Sharing Python data between processes using mmap | schmichael's blog
    • Demonstrates how both scripts change the shared value
    • Note that here a temporary file is created as storage for saved data - mmap is just a special interface for accessing this temporary file
    • multiprocessing.Process 下的 Python multiprocessing RemoteManager - SyncManager 的工作示例(通过 manager.start()) 与共享 Queue;服务器写入，客户端读取(共享数据)
    • 多处理模块和pyro的比较? - BaseManager 的工作示例(通过 server.serve_forever()) 与共享自定义类；服务器写入，客户端读取和写入
    • 如何将 python dict 与多处理同步 - 这个答案对 multiprocessing 陷阱，是 SyncManager(通过 manager.start())与共享字典的工作示例；服务器什么都不做，客户端读写
    • Python multiprocessing RemoteManager under a multiprocessing.Process - working example of SyncManager (via manager.start()) with shared Queue; server(s) writes, clients read (shared data)
    • Comparison of the multiprocessing module and pyro? - working example of BaseManager (via server.serve_forever()) with shared custom class; server writes, client reads and writes
    • How to synchronize a python dict with multiprocessing - this answer has a great explanation of multiprocessing pitfalls, and is a working example of SyncManager (via manager.start()) with shared dict; server does nothing, client reads and writes

    多亏了这些例子，我想出了一个例子，它本质上与 mmap 例子一样，方法来自synchronize a python dict"例子- 使用BaseManager(通过manager.start() 通过文件路径地址)和共享列表；服务器和客户端读取和写入(粘贴如下).请注意:

    Thanks to these examples, I came up with an example, which essentially does the same as the mmap example, with approaches from the "synchronize a python dict" example - using BaseManager (via manager.start() through file path address) with shared list; both server and client read and write (pasted below). Note that:

    • multiprocessing 管理器可以通过 manager.start() 或 server.serve_forever() 启动
      • serve_forever() 锁 - start() 没有
      • multiprocessing 中有自动记录功能:它似乎在 start()ed 进程中工作正常 - 但似乎忽略了 serve_forever()
      • multiprocessing managers can be started either via manager.start() or server.serve_forever()
        • serve_forever() locks - start() doesn't
        • There is auto-logging facility in multiprocessing: it seems to work fine with start()ed processes - but seems to ignore the ones that serve_forever()
        • 大多数示例使用 multiprocessing.Manager() - 这只是一个返回 SyncManager 的函数(不是 类实例化)，它是BaseManager 的特殊子类；并使用 start() - 但 not 用于独立运行的脚本之间的 IPC；这里使用了文件路径
        • 其他几个例子 serve_forever() 在独立运行的脚本之间使用 IPC 方法；此处使用 IP/套接字地址
        • 如果未指定地址，则会自动使用临时文件路径(参见 16.6.2.12. 记录有关如何查看此内容的示例)
        • Most examples use multiprocessing.Manager() - this is just a function (not class instantiation) which returns a SyncManager, which is a special subclass of BaseManager; and uses start() - but not for IPC between independently ran scripts; here a file path is used
        • Few other examples serve_forever() approach for IPC between independently ran scripts; here IP/socket address is used
        • If an address is not specified, then an temp file path is used automatically (see 16.6.2.12. Logging for an example of how to see this)

        除了同步python dict"一文中的所有陷阱之外，在列表的情况下还有其他陷阱.那个帖子注释:

        In addition to all the pitfalls in the "synchronize a python dict" post, there are additional ones in case of a list. That post notes:

        dict 的所有操作都必须使用方法而不是 dict 赋值来完成(syncdict["blast"] = 2 会因为 multiprocessing 共享自定义对象的方式而惨遭失败)

        All manipulations of the dict must be done with methods and not dict assignments (syncdict["blast"] = 2 will fail miserably because of the way multiprocessing shares custom objects)

        dict['key'] 获取和设置的解决方法是使用 dict 公共方法 get 和 更新.问题是没有这样的公共方法可以替代 list[index];因此，对于共享列表，我们还必须将 __getitem__ 和 __setitem__ 方法(对于 list 是私有的)注册为 exposed，这意味着我们还必须为 list 以及 :/

        The workaround to dict['key'] getting and setting, is the use of the dict public methods get and update. The problem is that there are no such public methods as alternative for list[index]; thus, for a shared list, in addition we have to register __getitem__ and __setitem__ methods (which are private for list) as exposed, which means we also have to re-register all the public methods for list as well :/

        嗯，我认为这些是最关键的；这是两个脚本 - 它们可以在单独的终端中运行(服务器优先)；在 Linux 上使用 Python 2.7 开发的注释:

        Well, I think those were the most critical things; these are the two scripts - they can just be ran in separate terminals (server first); note developed on Linux with Python 2.7:

        a.py(服务器):

        import multiprocessing
        import multiprocessing.managers
        
        import logging
        logger = multiprocessing.log_to_stderr()
        logger.setLevel(logging.INFO)
        
        
        class MyListManager(multiprocessing.managers.BaseManager):
            pass
        
        
        syncarr = []
        def get_arr():
            return syncarr
        
        def main():
        
            # print dir([]) # cannot do `exposed = dir([])`!! manually:
            MyListManager.register("syncarr", get_arr, exposed=['__getitem__', '__setitem__', '__str__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'])
        
            manager = MyListManager(address=('/tmp/mypipe'), authkey='')
            manager.start()
        
            # we don't use the same name as `syncarr` here (although we could);
            # just to see that `syncarr_tmp` is actually <AutoProxy[syncarr] object>
            # so we also have to expose `__str__` method in order to print its list values!
            syncarr_tmp = manager.syncarr()
            print("syncarr (master):", syncarr, "syncarr_tmp:", syncarr_tmp)
            print("syncarr initial:", syncarr_tmp.__str__())
        
            syncarr_tmp.append(140)
            syncarr_tmp.append("hello")
        
            print("syncarr set:", str(syncarr_tmp))
        
            raw_input('Now run b.py and press ENTER')
        
            print
            print 'Changing [0]'
            syncarr_tmp.__setitem__(0, 250)
        
            print 'Changing [1]'
            syncarr_tmp.__setitem__(1, "foo")
        
            new_i = raw_input('Enter a new int value for [0]: ')
            syncarr_tmp.__setitem__(0, int(new_i))
        
            raw_input("Press any key (NOT Ctrl-C!) to kill server (but kill client first)".center(50, "-"))
            manager.shutdown()
        
        if __name__ == '__main__':
          main()
        

        b.py(客户端)

        import time
        
        import multiprocessing
        import multiprocessing.managers
        
        import logging
        logger = multiprocessing.log_to_stderr()
        logger.setLevel(logging.INFO)
        
        
        class MyListManager(multiprocessing.managers.BaseManager):
            pass
        
        MyListManager.register("syncarr")
        
        def main():
          manager = MyListManager(address=('/tmp/mypipe'), authkey='')
          manager.connect()
          syncarr = manager.syncarr()
        
          print "arr = %s" % (dir(syncarr))
        
          # note here we need not bother with __str__ 
          # syncarr can be printed as a list without a problem:
          print "List at start:", syncarr
          print "Changing from client"
          syncarr.append(30)
          print "List now:", syncarr
        
          o0 = None
          o1 = None
        
          while 1:
            new_0 = syncarr.__getitem__(0) # syncarr[0]
            new_1 = syncarr.__getitem__(1) # syncarr[1]
        
            if o0 != new_0 or o1 != new_1:
              print 'o0: %s => %s' % (str(o0), str(new_0))
              print 'o1: %s => %s' % (str(o1), str(new_1))
              print "List is:", syncarr
        
              print 'Press Ctrl-C to exit'
              o0 = new_0
              o1 = new_1
        
            time.sleep(1)
        
        
        if __name__ == '__main__':
            main()
        

        最后要说的是，在 Linux 上创建了 /tmp/mypipe - 但它是 0 字节，并且具有属性 srwxr-xr-x(用于套接字)；我想这让我很高兴，因为我既不必担心网络端口，也不必担心诸如 :)

        As a final remark, on Linux /tmp/mypipe is created - but is 0 bytes, and has attributes srwxr-xr-x (for a socket); I guess this makes me happy, as I neither have to worry about network ports, nor about temporary files as such :)

        其他相关问题:

        • Python:可以在 2 个独立进程之间共享内存数据(很好的解释)
        • 高效的 Python 到 Python IPC
        • Python:将变量发送到另一个脚本
        • Python: Possible to share in-memory data between 2 separate processes (very good explanation)
        • Efficient Python to Python IPC
        • Python: Sending a variable to another script

        这篇关于如何在python中跨脚本共享变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！