在EC2主节点上初始化Ray时出错 [英] Error while initializing Ray on an EC2 master node

问题描述

我正在使用Ray在AWS EC2的Ubuntu 14.04集群上运行并行循环.以下Python 3脚本在只有4个工作线程的本地计算机上运行良好(导入和本地初始化被忽略):-

I am using Ray to run a parallel loop on an Ubuntu 14.04 cluster on AWS EC2. The following Python 3 script works well on my local machine with just 4 workers (imports and local initializations left out):-

ray.init()           #initialize Ray

@ray.remote
def test_loop(n):
    c=tests[n,0]                            
    tout=100                
    rc=-1   

    with tmp.TemporaryDirectory() as path: #Create a temporary directory        
        for files in filelist:        #then copy in all of the 
            sh.copy(filelist,path)    #files
        txtfile=path+'/inputf.txt'    #create the external
        fileId=open(txtfile,'w')      #data input text file,
        s='Number = '+str(c)+"\n"     #write test number,           
        fileId.write(s)
        fileId.close()                #close external parameter file,
        os.chdir(path)                #and change working directory

        try:                                    #Try running simulation:
            rc=sp.call('./simulation.run',timeout=tout,stdout=sp.DEVNULL,\
        stderr=sp.DEVNULL,shell=True)           #(must use .call for timeout)
            outdat=sio.loadmat('outputf.dat')   #get the output data struct
            rt_Data=outdat.get('rt_Data')       #extract simulation output
            err=float(rt_Data[-1])              #use final value of error
        except:                                 #If system fails to execute,
            err=deferr                          #use failure default 
        #end try

        if (err<=0) or (err>deferr) or (rc!=0): 
            err=deferr                          #Catch other types of failure
    return err 

if __name__=='__main__':
    result=ray.get([test_loop.remote(n) for n in range(0,ntest)])
    print(result)

这里不寻常的一点是，simulation.run在运行时必须从外部文本文件中读取不同的测试号.循环的所有迭代的文件名都相同，但是测试号不同.

The unusual bit here is that the simulation.run has to read in a different test number from an external text file when it runs. The file name is the same for all iterations of the loop, but the test number is different.

我使用Ray启动了EC2集群，可用的CPU数量等于n(我相信Ray不会默认为多线程).然后，我不得不使用rsync将文件列表(包括Python脚本)从本地计算机复制到主节点，因为我无法从配置中执行此操作(请参阅最近的问题:"Ray不会在EC2上启动工作人员" ).然后ssh进入该节点，并运行脚本.结果是文件查找错误:-

I launched an EC2 cluster using Ray, with the number of CPUs available equal to n (I am trusting that Ray will not default to multi-threading). Then I had to copy the filelist (which includes the Python script) from my local machine to the master node using rsync, because I couldn't do this from the config (see recent question: "Workers not being launched on EC2 by Ray"). Then ssh into that node, and run the script. The result is a file-finding error:-

~$ python3 test_small.py
2019-04-29 23:39:27,065 WARNING worker.py:1337 -- WARNING: Not updating worker name since `setproctitle` is not installed. Install this with `pip install setproctitle` (or ray[debug]) to enable monitoring of worker processes.
2019-04-29 23:39:27,065 INFO node.py:469 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-29_23-39-27_3897/logs.
2019-04-29 23:39:27,172 INFO services.py:407 -- Waiting for redis server at 127.0.0.1:42930 to respond...
2019-04-29 23:39:27,281 INFO services.py:407 -- Waiting for redis server at 127.0.0.1:47779 to respond...
2019-04-29 23:39:27,282 INFO services.py:804 -- Starting Redis shard with 0.21 GB max memory.
2019-04-29 23:39:27,296 INFO node.py:483 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-29_23-39-27_3897/logs.
2019-04-29 23:39:27,296 INFO services.py:1427 -- Starting the Plasma object store with 0.31 GB memory using /dev/shm.
(pid=3917) sh: 0: getcwd() failed: No such file or directory
    2019-04-29 23:39:44,960 ERROR worker.py:1672 -- Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 909, in _process_task
self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 820, in _store_outputs_in_object_store
self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 375, in put_object
self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 309, in store_and_register
self.task_driver_id))
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 238, in get_serialization_context
_initialize_serialization(driver_id)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 1148, in _initialize_serialization
serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 326, in default_serialization_context
register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 321, in register_default_serialization_handlers
_register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 129, in _register_custom_pandas_handlers
import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/__init__.py", line 42, in <module>
from pandas.core.api import *
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module>
from pandas.core.groupby import Grouper
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/groupby.py", line 49, in <module>
from pandas.core.frame import DataFrame
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py", line 74, in <module>
from pandas.core.series import Series
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/series.py", line 3042, in <module>
import pandas.plotting._core as _gfx  # noqa
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/plotting/__init__.py", line 8, in <module>
from pandas.plotting import _converter
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/plotting/_converter.py", line 7, in <module>
import matplotlib.units as units
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py", line 1060, in <module>
rcParams = rc_params()
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py", line 892, in rc_params
fname = matplotlib_fname()
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py", line 736, in matplotlib_fname
for fname in gen_candidates():
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py", line 725, in gen_candidates
yield os.path.join(six.moves.getcwd(), 'matplotlibrc')
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

然后问题似乎在所有其他工人身上重复出现，并最终放弃:-

The problem then seems to repeat for all the other workers and finally gives up:-

AttributeError: module 'pandas' has no attribute 'core'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-04-29 23:44:08,489 ERROR worker.py:1672 -- A worker died or was killed while executing task 000000002d95245f833cdbf259672412d8455d89.
Traceback (most recent call last):
  File "test_small.py", line 82, in <module>
result=ray.get([test_loop.remote(n) for n in range(0,ntest)])
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 2184, in get
raise value
ray.exceptions.RayWorkerError: The worker died unexpectedly while executing this task.

我怀疑我没有正确初始化Ray.我尝试使用ray.init(redis_address ="172.31.50.149:6379")-这是形成集群时提供的redis地址，但错误或多或少都相同.我还尝试在主服务器上启动Ray(以防需要启动):-

I suspect that I am not initializing Ray correctly. I tried with ray.init(redis_address="172.31.50.149:6379") - which was the redis address given when the cluster was formed, but the error was more or less the same. I also tried starting Ray on the master (in case it needed starting):-

~$ ray start --redis-address 172.31.50.149:6379 #Start Ray
2019-04-29 23:46:20,774 INFO services.py:407 -- Waiting for redis server at 172.31.50.149:6379 to respond...
2019-04-29 23:48:29,076 INFO services.py:412 -- Failed to connect to the redis server, retrying.

.... etc.

推荐答案

在主节点上安装pandas和matplotlib似乎已解决了该问题. Ray现在可以成功初始化.

The installation of pandas and matplotlib on the master node seems to have solved the problem. Ray now initializes successfully.

这篇关于在EC2主节点上初始化Ray时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！