我怎样才能使我的python代码运行得更快 [英] How can I make my python code run faster
问题描述
我正在研究循环多个netcdf文件(约28G)的代码. netcdf文件在整个域中具有多个4D变量[时间，东西，南北，高度].目标是遍历这些文件，并遍历域中所有这些变量的每个位置，并提取某些变量以存储到大数组中.当文件丢失或不完整时，我用99.99填充值.现在，我只是通过循环2个每日的netcdf文件进行测试，但是由于某种原因，它要花很多时间(〜14小时).我不确定是否有一种方法可以优化此代码.我认为python不需要花这么长时间来完成此任务，但也许是python或我的代码有问题.下面是我的代码，希望它是可读的，并且对如何使此方法更快的任何建议深表感谢:
I am working on code that loops over multiple netcdf files (large ~28G). The netcdf files have multiple 4D variables[time, east-west, south-north, height] throughout a domain. The goal is to loop over these files and to loop over each location of all of these variables in the domain and pull certain variables to store into a large array. When there is missing or incomplete files I fill the values with 99.99. Right now I am just testing by looping over 2 daily netcdf files but for some reason it is taking forever (~14 hours). I am not sure if there is a way to optimize this code. I don't think that python should take this long for this task but maybe it is a problem with python or my code. Below is my code hopefully it is readable and any suggestions on how to make this faster is greatly appreciated:
#Domain to loop over
k_space = np.arange(0,37)
j_space = np.arange(80,170)
i_space = np.arange(200,307)
predictors_wrf=[]
names_wrf=[]
counter = 0
cdate = start_date
while cdate <= end_date:
if cdate.month not in month_keep:
cdate+=inc
continue
yy = cdate.strftime('%Y')
mm = cdate.strftime('%m')
dd = cdate.strftime('%d')
filename = wrf_path+'\wrfoutRED_d01_'+yy+'-'+mm+'-'+dd+'_'+hour_str+'_00_00'
for i in i_space:
for j in j_space:
for k in k_space:
if os.path.isfile(filename):
f = nc.Dataset(filename,'r')
times = f.variables['Times'][1:]
num_lines = times.shape[0]
if num_lines == 144:
u = f.variables['U'][1:,k,j,i]
v = f.variables['V'][1:,k,j,i]
wspd = np.sqrt(u**2.+v**2.)
w = f.variables['W'][1:,k,j,i]
p = f.variables['P'][1:,k,j,i]
t = f.variables['T'][1:,k,j,i]
if num_lines < 144:
print "partial files for WRF: "+ filename
u = np.ones((144,))*99.99
v = np.ones((144,))*99.99
wspd = np.ones((144,))*99.99
w = np.ones((144,))*99.99
p = np.ones((144,))*99.99
t = np.ones((144,))*99.99
else:
u = np.ones((144,))*99.99
v = np.ones((144,))*99.99
wspd = np.ones((144,))*99.99
w = np.ones((144,))*99.99
p = np.ones((144,))*99.99
t = np.ones((144,))*99.99
counter=counter+1
predictors_wrf.append(u)
predictors_wrf.append(v)
predictors_wrf.append(wspd)
predictors_wrf.append(w)
predictors_wrf.append(p)
predictors_wrf.append(t)
u_names = 'u_'+str(k)+'_'+str(j)+'_'+str(i)
v_names = 'v_'+str(k)+'_'+str(j)+'_'+str(i)
wspd_names = 'wspd_'+str(k)+'_'+str(j)+'_'+str(i)
w_names = 'w_'+str(k)+'_'+str(j)+'_'+str(i)
p_names = 'p_'+str(k)+'_'+str(j)+'_'+str(i)
t_names = 't_'+str(k)+'_'+str(j)+'_'+str(i)
names_wrf.append(u_names)
names_wrf.append(v_names)
names_wrf.append(wspd_names)
names_wrf.append(w_names)
names_wrf.append(p_names)
names_wrf.append(t_names)
cdate+=inc
推荐答案
这是to脚的第一步，用来收紧您的forloop.由于每个文件仅使用一次文件形状，因此可以将处理移到循环外，这样可以减少中断处理时的数据加载量.我仍然不了解counter和inc的作用，因为它们似乎没有在循环中更新.您肯定想研究重复的字符串连接性能，或将predictors_wrf和names_wrf追加的性能如何作为起点
This is a lame first pass to tighten up your forloops. Since you only use the file shape once per file, you can move the handling outside the loop which should reduce the amount of loading of data in interrupting processing. I still don't get what counter and inc do as they don't seem to be updated in the loop. You definitely want to look into repeated string concatenation performance, or how the performance of your appending to predictors_wrf and names_wrf looks as starting points
k_space = np.arange(0,37)
j_space = np.arange(80,170)
i_space = np.arange(200,307)
predictors_wrf=[]
names_wrf=[]
counter = 0
cdate = start_date
while cdate <= end_date:
if cdate.month not in month_keep:
cdate+=inc
continue
yy = cdate.strftime('%Y')
mm = cdate.strftime('%m')
dd = cdate.strftime('%d')
filename = wrf_path+'\wrfoutRED_d01_'+yy+'-'+mm+'-'+dd+'_'+hour_str+'_00_00'
file_exists = os.path.isfile(filename)
if file_exists:
f = nc.Dataset(filename,'r')
times = f.variables['Times'][1:]
num_lines = times.shape[0]
for i in i_space:
for j in j_space:
for k in k_space:
if file_exists:
if num_lines == 144:
u = f.variables['U'][1:,k,j,i]
v = f.variables['V'][1:,k,j,i]
wspd = np.sqrt(u**2.+v**2.)
w = f.variables['W'][1:,k,j,i]
p = f.variables['P'][1:,k,j,i]
t = f.variables['T'][1:,k,j,i]
if num_lines < 144:
print "partial files for WRF: "+ filename
u = np.ones((144,))*99.99
v = np.ones((144,))*99.99
wspd = np.ones((144,))*99.99
w = np.ones((144,))*99.99
p = np.ones((144,))*99.99
t = np.ones((144,))*99.99
else:
u = np.ones((144,))*99.99
v = np.ones((144,))*99.99
wspd = np.ones((144,))*99.99
w = np.ones((144,))*99.99
p = np.ones((144,))*99.99
t = np.ones((144,))*99.99
counter=counter+1
predictors_wrf.append(u)
predictors_wrf.append(v)
predictors_wrf.append(wspd)
predictors_wrf.append(w)
predictors_wrf.append(p)
predictors_wrf.append(t)
u_names = 'u_'+str(k)+'_'+str(j)+'_'+str(i)
v_names = 'v_'+str(k)+'_'+str(j)+'_'+str(i)
wspd_names = 'wspd_'+str(k)+'_'+str(j)+'_'+str(i)
w_names = 'w_'+str(k)+'_'+str(j)+'_'+str(i)
p_names = 'p_'+str(k)+'_'+str(j)+'_'+str(i)
t_names = 't_'+str(k)+'_'+str(j)+'_'+str(i)
names_wrf.append(u_names)
names_wrf.append(v_names)
names_wrf.append(wspd_names)
names_wrf.append(w_names)
names_wrf.append(p_names)
names_wrf.append(t_names)
cdate+=inc
这篇关于我怎样才能使我的python代码运行得更快的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
